Work Paper
Joint IOU
Standard Performance Contract Program —
All Measures
Net-to-Gross Ratios
2
Joint Investor-Owned Utilities Workpaper May 16, 2008
At a Glance Summary
Measure Names / All MeasuresProgram Type / Retrofit
Market Segment / Nonresidential
Delivery Type / Customized Incentives, Performance Contracting
Net-to-Gross Ratios / 0.69
Important Comments
Note: The information provided in this work paper was developed using the best available technical resources at the time this document was prepared.
Table of Contents
At a Glance Summary 3
Table of Contents 4
List of Tables 4
Section 1. 5
1.1 General Measure and Program Description 5
1.2 DEER 2008 NTG Update Differences Analysis 5
1.3 EM&V, Market Potential, and Other Studies 5
Total Project Savings. 5
Adjustment for Self-Report Methodology Bias. 6
Section 2. Calculation Methods 6
2.1 NTG Estimation Methodology 6
Net-to-Gross Ratios for Different Program Strategies 6
Section 3. Summary 7
References 7
List of Tables
Table 1. 7
2
Joint Investor-Owned Utilities Workpaper May 16, 2008
Section 1.
1.1 General Measure and Program Description
This work paper documents the net-to-gross ratio (NTGR) for all measure categories in the 2006-2008 nonresidential Standard Performance Contract (SPC) program. The SPC program offers customized incentives that are calculated on the basis of project performance related to energy efficiency for equipment retrofits or replacements.
1.2 DEER 2008 NTG Update Differences Analysis
For the NTGRs for this program, the 2008 update to the Database for Energy Efficiency Resources (DEER)¹ draws on the recent measurement and evaluation study² of the 2004-2005 statewide nonresidential Standard Performance Contract Program. That study reports an overall NTGR of 0.69. The DEER Update proposes a value of 0.54. The difference lies in the disregard for adjustments to the raw measurement of the net-to-gross ratio.
1.3 EM&V, Market Potential, and Other Studies
In the 2004-2005 statewide nonresidential Standard Performance Contract Program evaluation study², the independent evaluation contractor calculated a net-to-gross ratio of 0.69 for the program.
Total Project Savings.
The unit of analysis in the SPC program should be the project, not a large number of separate measures. The term “participant spillover” has been used inaccurately in the assessment of net savings of the Standard Performance Contract programs, resulting in the inappropriate reduction of the NTGR value by 0.05 in the DEER Update.
The DEER Update NTGR excludes savings from non-incented energy efficiency measures and practices that are part of an SPC project. Customers do file applications for SPC projects that include measures for which incentives are not available. These measures are undertaken as part of the SPC project and do result in savings. In addition, during the course of the project, the customer may enlarge the project’s scope and not bother to change the application because the project as a whole, with the project incentive offered, can still meet their financial criteria. In the loose language that has grown common among some California evaluations of calculated savings programs in the last few years, these measures have inappropriately been labeled “participant spillover.”
Spillover is defined in the California Energy Efficiency Evaluation Protocols³ as “reductions in energy consumption and/or demand in a utility’s service area caused by the presence of the DSM program, beyond program related gross or net savings of participants” (emphasis added). Participant spillover is then the additional energy savings achieved from some separate activity that the customer later undertakes as a result of learning about energy efficiency through their initial project participation. When evaluating a prescriptive rebate program, a customer’s subsequent purchase of additional measures or different measures outside the program is appropriately identified as participant spillover. But in a calculated savings program, the whole project or building is the unit of measurement. The net-to-gross ratio should be focused on identifying the savings that were “program related,” that is, related to the project that the SPC program encouraged.
By estimating project savings only as incented-measure savings, this inappropriate methodology ignores one of the major arguments for customized project-level programs — they reduce lost opportunities and help customers to undertake more comprehensive retrofits, achieving greater savings from more measures than a prescriptive widget-by-widget approach. The project level incentive allows customers to put together a project that “pencils out” for them as a whole, even though some parts of the project will not be eligible for financial incentives from an energy efficiency program.
This perspective is consistent with the Commission’s direction that spillover impacts should not be counted as part of the goal achievement for 2006-08 programs. The rationale for that directive is that spillover savings do not occur as part of program participation and must therefore have inherently greater uncertainty of measurement. What has been incorrectly labeled “participant spillover” in the SPC program evaluation does not fit this rationale.
Adjustment for Self-Report Methodology Bias.
Self-report bias is a widely recognized flaw of the customer decision-maker survey approach to estimating freeridership. In the research plan for the evaluation of the 2004-05 SPC Program, Itron proposed, and the ED and the study advisory committee approved, to measure freeridership using a survey-based measurement tool with an accepted self-report bias adjustment. This same approach has used in previous SPC evaluations. By understanding the bias of the self-report approach and correcting for it (see XENERGY, 2001), the self-report approach’s measurement accuracy can be improved the same way that a rifle’s sights can be made more accurate by adjusting for windage. Ignoring this measurement correction is simply poor research technique.
This measurement bias, like any other measurement bias, should be corrected, especially when it is measurable. The key factor in this instance is that it has been measured in a previous study; other programs may have not been able to measure it or did not make any attempt to measure it. For large nonresidential programs where multiple decisions makers are involved, such measurement bias exists and needs to be identified and corrected.
Section 2. Calculation Methods
2.1 NTG Estimation Methodology
Net-to-Gross Ratios for Different Program Strategies
The applicable net-to-gross ratio for this measure is 0.69, based on the evaluation study for the 2004-05 program. The self-reported survey-based estimate of 0.54 is adjusted by adding 0.10 to offset the downward bias that was recommended in the XENERGY study. It is further adjusted by adding 0.05 for the technique’s underreporting of total project savings. Thus:
SPC NTGR = 0.54 + 0.10 + 0.05 = 0.69
Section 3. Summary
Table 1. Net to Gross Ratio (NTGR) Values
Program Approach / NTGRStandard Performance Contract / 0.69
References
1. Database for Energy Efficient Resources (DEER), DEER Net-to-Gross Update 2008,
May 2008
2. 2004-2005 Statewide Nonresidential Standard Performance Contract Program Measurement and Evaluation Study: Impact, Process and Market Evaluation – Final Report. Itron, March 19, 2008 (First Draft).
3. California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. The TecMarket Works Team, April 2006, p. 241.
2
Joint Investor-Owned Utilities Workpaper May 16, 2008