2013-2015 Local Government Partnership Programs Impact Evaluation Research Plan
Prepared for the Energy Division of the
California Public Utilities Commission
Itron, Inc.
12348 High Bluff Drive, Suite 210
San Diego, California 92130
(858) 724-2620
July 14, 2016
Itron, Inc. 1-2 Introduction
2013-2015 LGP Impact Evaluation Research Plan
Table of Contents
1 Introduction 1-1
1.1 Goals and Objectives 1-1
1.2 Overview of Programs to be Studied 1-2
1.3 Evaluator Contact Information 1-5
2 Evaluation Approach 2-1
2.1 Net-to-Gross Analysis 2-1
2.2 Ex Post Savings and Cost Effectiveness Assessment 2-2
3 Data Sources and Sample Design 3-1
3.1 Key Existing Data Sources 3-1
3.1.1 Program Tracking Data 3-1
3.1.2 2013-2015 ESPI Impact Evaluation Results 3-1
3.1.3 Comprehensiveness Analysis Study Results 3-1
3.1.4 CIS Billing Data 3-1
3.2 NTG Telephone Survey Sample Design 3-2
4 Work Plan, Timeline and Budget 4-1
4.1 Work plan 4-1
4.1.1 Task 1: Project Management 4-1
4.1.2 Task 2: Stakeholder Engagement 4-1
4.1.3 Task 3: Develop Detailed Research Plan 4-1
4.1.4 Task 4: Tracking System Management and Sampling 4-2
4.1.5 Task 5: Telephone Survey Implementation 4-2
4.1.6 Task 6: Analysis 4-2
4.1.7 Task 7: Reporting, Recommendations, and Comment Response 4-2
4.2 Timeline 4-3
4.3 Budget 4-5
List of Figures
Figure 41: LGP Impact Evaluation Timeline 4-4
List of Tables
Table 11: 2013-2015 Ex Ante Lifecycle Savings for PG&E’s LGP Resource Programs 1-3
Table 12: 2013-2015 Ex Ante Lifecycle Savings for SCE’s LGP Resource Programs 1-4
Table 31: Mapping of PG&E’s LGP Programs to Net Program Groups 3-2
Table 32: Mapping of SCE’s LGP Programs to Net Program Groups 3-3
Table 33: NTGR Telephone Survey Sample Design by Net Program Group with Expected Relative Precisions 3-5
Table 41: Budget by Task and Evaluation Period 4-5
Itron, Inc. ii Table of Contents
2013-2015 LGP Impact Evaluation Research Plan
1Introduction
1.1 Goals and Objectives
This document presents the Research Plan for the impact evaluation of the 2013-2015 Local Government Partnership (LGP) programs offered by Pacific Gas and Electric (PG&E) and Southern California Edison (SCE). There are two overarching goals for this study:
1. Perform a net-to-gross (NTG) analysis on the 2013-2015 LGP program participants, estimating ex post net-to-gross ratios (NTGRs) and investigating reasons for free ridership, and
2. Utilize the NTGRs from this study and existing ex post gross savings values from recent CPUC managed impact evaluations to update net ex post savings values and develop estimates of key cost-effectiveness metrics for each of the 2013-2015 LGP resource programs.
A number of research questions will be addressed by this evaluation that are related to each of these two study goals.
Related NTG analysis research questions include the following.
n What is the overall NTGR at the program administrator (PA) level for PG&E and SCE’s LGP resource programs over the 2013-2015 timeframe?[1]
n What is the NTGR for a number of large individual LGP resource programs or groups of smaller programs?
n Which key LGP programs or groups of programs have the lowest rates of free ridership?
n What are the most influential program and non-program factors in the customer’s decision to install the rebated equipment for key LGP programs and groups of programs?
n For each key LGP program or group of programs, what percent of installed equipment was influenced by the program to perform an early replacement as opposed to a replacement on burnout?
n Which key LGP programs or group of programs was most successful in influencing early replacement installations?
Related ex post savings and cost effectiveness research questions include the following:
n What are the aggregate first year and lifecycle ex post gross and net savings for each PG&E and SCE LGP resource program over the 2013-2015 timeframe?
n What are the overall net and gross realization rates (NRRs and GRRs) separately for each PG&E and SCE LGP resource program, and overall for PG&E’s and SCE’s LGP sector?
n Which measures or classes of measures are driving realization rates to differ from 100%?
n What are the Total Resource Cost (TRC) and Program Administrator Cost (PAC) values for each PG&E and SCE LGP resource program based on the ex post NTGRs and savings values?
n How do the ex post TRC/PAC values compare to those reported (i.e., ex ante) by SCE and PG&E for each LGP resource program?
n Which LGP resource programs are most cost-effective with respect to the TRC and PAC metrics?
n Which LGP resource programs are more successful in achieving deeper savings, and doing so more cost effectively?
1.2 Overview of Programs to be Studied
This evaluation will address all LGP resource programs offered by PG&E and SCE during the 2013-2015 timeframe (no resource programs were offered by SDG&E or SCG). Table 11 and Table 12 below list each of the LGP resource programs offered by PG&E and SCE, along with their corresponding 2013-2015 ex ante lifecycle kW, kWh, and therm savings, shown as both a magnitude and a percentage of each PA’s total LGP ex ante claim.
Table 11: 2013-2015 Ex Ante Lifecycle Savings for PG&E’s LGP Resource Programs
Program ID / Program Name / Site Count / 2013-2015 Ex Ante Net Lifecycle SavingsTOTAL / Percent of PA-LGP Pop
MW / MWh / Therm / kW / kWh / Therm
PGE2110051 / LGEAR / 6,988 / 40 / 328,961 / 104,519 / 17% / 17% / 2%
PGE211007 / AMBAG / 1,296 / 25 / 174,549 / 396,258 / 10% / 9% / 9%
PGE211009 / EAST BAY / 8,650 / 25 / 326,260 / 886,901 / 11% / 17% / 19%
PGE211010 / FRESNO / 1,065 / 25 / 142,373 / 227,166 / 11% / 8% / 5%
PGE211011 / KERN / 854 / 20 / 121,436 / 38,051 / 8% / 6% / 1%
PGE211012 / MADERA / 158 / 3 / 14,751 / 202 / 1% / 1% / 0%
PGE211013 / MARIN COUNTY / 1,948 / 4 / 32,856 / 898,927 / 2% / 2% / 19%
PGE211014 / MENDOCINO COUNTY / 83 / 2 / 17,032 / 10,605 / 1% / 1% / 0%
PGE211015 / NAPA COUNTY / 245 / 3 / 22,318 / 19,566 / 1% / 1% / 0%
PGE211016 / REDWOOD COAST / 1,394 / 5 / 27,628 / 30,079 / 2% / 1% / 1%
PGE211018 / SAN LUIS OBISPO COUNTY / 548 / 6 / 41,240 / 1,150 / 2% / 2% / 0%
PGE211019 / SAN MATEO COUNTY / 842 / 11 / 99,556 / 184,865 / 5% / 5% / 4%
PGE211020 / SANTA BARBARA / 475 / 6 / 49,216 / 172 / 3% / 3% / 0%
PGE211021 / SIERRA NEVADA / 808 / 12 / 83,757 / 661 / 5% / 4% / 0%
PGE211022 / SONOMA COUNTY / 1,043 / 10 / 73,628 / 238,192 / 4% / 4% / 5%
PGE211023 / SILICON VALLEY / 1,516 / 17 / 174,321 / 442,162 / 7% / 9% / 10%
PGE211024 / SAN FRANCISCO / 1,139 / 27 / 151,569 / 1,164,324 / 11% / 8% / 25%
Table 12: 2013-2015 Ex Ante Lifecycle Savings for SCE’s LGP Resource Programs
Program ID / Program Name / Site Count / 2013-2015 Ex Ante Net Lifecycle SavingsTOTAL / Percent of PA-LGP Pop
MW / MWh / Therm / kW / kWh / Therm
SCE-13-L-002-1 / HIGH DESERT REGIONAL ELP / 24 / 0 / 861 / - / 0% / 0% / 0%
SCE-13-L-002A / CITY OF BEAUMONT ELP / 4 / 0 / 355 / - / 0% / 0% / 0%
SCE-13-L-002B / CITY OF LONG BEACH ELP / 13 / 2 / 6,804 / - / 3% / 2% / 0%
SCE-13-L-002C / CITY OF REDLANDS ELP / 19 / 1 / 6,599 / - / 2% / 2% / 0%
SCE-13-L-002D / CITY OF SANTA ANA ELP / 278 / 1 / 16,455 / - / 2% / 4% / 0%
SCE-13-L-002E / CITY OF SIMI VALLEY ELP / 7 / 0 / 845 / - / 0% / 0% / 0%
SCE-13-L-002F / GATEWAY CITIES ELP / 52 / 1 / 9,537 / - / 1% / 2% / 0%
SCE-13-L-002G / COMMUNITY ELP / 196 / 7 / 56,368 / 20,997 / 13% / 14% / 3%
SCE-13-L-002H / EASTERN SIERRA ELP / 28 / 0 / 836 / - / 0% / 0% / 0%
SCE-13-L-002J / DESERT CITIES ELP / 319 / 4 / 25,861 / - / 7% / 7% / 0%
SCE-13-L-002K / KERN COUNTY ELP / 66 / 1 / 4,279 / - / 2% / 1% / 0%
SCE-13-L-002L / ORANGE COUNTY CITIES ELP / 51 / 6 / 28,609 / 27,238 / 10% / 7% / 4%
SCE-13-L-002M / SAN GABRIEL VALLEY ELP / 127 / 3 / 29,999 / - / 6% / 8% / 0%
SCE-13-L-002N / SAN JOAQUIN VALLEY ELP / 118 / 3 / 19,032 / 1,629 / 6% / 5% / 0%
SCE-13-L-002O / SOUTH BAY ELP / 187 / 3 / 19,520 / 9 / 5% / 5% / 0%
SCE-13-L-002P / SOUTH SANTA BARBARA COUNTY ELP / 73 / 2 / 10,906 / - / 4% / 3% / 0%
SCE-13-L-002Q / VENTURA COUNTY ELP / 290 / 4 / 32,888 / 2,412 / 8% / 8% / 0%
SCE-13-L-002R / WESTERN RIVERSIDE ELP / 54 / 1 / 5,853 / - / 2% / 1% / 0%
SCE-13-L-002S / CITY OF ADELANTO ELP / 8 / 2 / 20,558 / - / 4% / 5% / 0%
SCE-13-L-002T / WEST SIDE ELP / 17 / 1 / 5,369 / - / 2% / 1% / 0%
SCE-13-L-003C / COUNTY OF LOS ANGELES EEP / 224 / 9 / 68,332 / 672,042 / 16% / 18% / 91%
SCE-13-L-003D / COUNTY OF RIVERSIDE EEP / 10 / 1 / 7,112 / 5,558 / 2% / 2% / 1%
SCE-13-L-003E / COUNTY OF SAN BERNARDINO EEP / 57 / 3 / 13,210 / 10,744 / 6% / 3% / 1%
This research plan includes the evaluation goals and objectives, the researchable issues, information on the programs and measures included in the evaluation, data sources that will be used, the sampling approach, the methods by which these measures will be evaluated, and the timeline for the evaluation.
1.3 Evaluator Contact Information
Itron will serve as the Prime Contractor managing this study, led by Mr. John Cavalli. The ED Project Manager for this study is Mr. Jeremy Battis.
The following is Mr. Battis’ and Mr. Cavalli’s contact information:
Firm / Lead / Contact InformationCPUC
505 Van Ness Avenue
San Francisco, CA 94102 / Jeremy Battis,
Energy Division
Local Government Programs and Regional Initiatives Statewide Lead Analyst / Phone: (415) 703-3041
Email:
Itron, Inc.
1111 Broadway, Suite 1800
Oakland, CA 94607 / John Cavalli,
Director Consulting & Analysis / Phone: (510) 844-2876
Email:
1-2 Introduction
2013-2015 LGP Impact Evaluation Research Plan
2Evaluation Approach
This impact evaluation of the LGP resource programs will consist of two distinct components, a NTG analysis and an ex post savings and cost-effectiveness assessment, each of which is discussed in detail in this section.
2.1 Net-to-Gross Analysis
The primary objective of the NTG analysis is to develop NTGRs for a select number of LGP programs and program groups, as well as by PA, for the 2013-2015 timeframe. This analysis will rely on existing survey data collection from the 2013-2015 nonresidential efficiency savings and performance incentive (ESPI)[2] impact evaluations, supplemented with new data collection gathered under this study.[3] The survey battery and analysis approach for estimating NTGRs will rely on the existing NTGR approach used for the 2013-2015 nonresidential deemed ESPI impact evaluation studies.[4] Only free ridership will be addressed by this analysis as a separate cross-cutting study, the 2013-2014 Nonresidential Spillover Study, is evaluating the spillover component of net. NTGR results will be compared across the selected programs and program groups to identify those with the lowest rates of free ridership. Comparisons at the PA level for all resource LGP programs will also be compared to similar NTG results for other sectors, such as Deemed, Direct Install, Third Party, and Custom programs by utilizing results from the 2013-2015 ESPI impact evaluations.
The NTGR survey battery contains a number of questions aimed at understanding how various program and non-program factors may have influenced the participants to install their rebated equipment under the LGP programs. An analysis of the underlying survey questions that comprise the NTGR will be performed to identify the most influential program and non-program factors in the customer’s decision to install the rebated equipment for key LGP programs and groups of programs. Other survey questions will also be analyzed in an attempt to better understand the drivers of free ridership.
Another component of the NTG survey is a battery of questions that is utilized to estimate if the program influenced the customer to make an early replacement, or if the equipment was replaced on burnout. For relevant equipment (such as linear fluorescent fixtures), an early replacement analysis will be conducted to estimate what percent of installed equipment was influenced by the program to perform an early replacement as opposed to a replacement on burnout. The team will also identify the key drivers differentiating an early replacement versus a replacement on burnout. The early replacement analysis will allow the team to identify which key LGP programs or groups of programs were most successful in influencing early replacement installations. Early replacement rates at the PA level for all non-resource LGP programs can also be compared to similar results for other sectors, such as Deemed, Direct Install, Third Party, and Custom programs by utilizing results from the 2013-2015 ESPI impact evaluations. It is important to note that only certain types of equipment are considered to feasibly have an early replacement. For example, CFLs replacing incandescents are only considered to be a replacement on burnout due to the short effective useful life of an incandescent. Therefore, the sample sizes for analyzing early replacement rates will be more limited. Because of this, the team may not be able to conduct this analysis at a fine level of segmentation, such as at the program level, so the analysis may be limited to comparisons at the PA level.