IOU Behavioral Programs Research Plan - Draft
Report No.: Work Orders ED-Res. 3.1 through 3.4
Document No.: [Document No.]
Date: July 22, 2016
CPUC 2015 Residential Roadmap Impact Evaluations
IOU Behavioral Programs Research Plan
Customer: / Peter Franzese, CPUC Energy Division
Contact person: / Valerie Richardson
July 22, 2016
[Report No.], [Rev.]
[Document No.]
Task 000 Research Plan
Task and objective:
Valerie Richardson
Prepared by: / Verified by: / Approved by:DNV GL – www.dnvgl.com
Table of contents
1 INTRODUCTION 1
1.1 Evaluation overview 1
1.2 Goals and objectives, including but not directly tied to load impact estimation 2
2 RESEARCHABLE ISSUES 3
2.1 PG&E HER Program 3
2.2 SCE HER AND HEES-Enhancement Programs 4
2.3 SDG&E HER and Manage-Act-Save Programs 5
2.4 MCE Home Utility Reports Program 8
2.5 Cost-Effectiveness 9
3 Applicable protocols 9
4 Discussion of how plan may need to evolve 10
5 Description of relevant measures and programs 12
6 Data sources Required 12
6.1 Key primary data sources 12
6.2 Likely secondary data sources 13
7 Evaluation Sample design and selection 13
8 Impact methodologies 14
8.1 Baseline Conditions 14
8.2 Gross demand impacts 14
8.3 Rigor 14
8.4 Adjusted energy impacts 15
8.5 EUL/RUL 15
9 Evaluation coordination efforts 15
10 Timeline 16
11 Work plan 16
Appendix A 18
11.1 Methodology: Energy Savings 18
DNV GL - www.dnvgl.com / Page i1 INTRODUCTION
1.1 Evaluation overview
The behavioral program initiatives in 2015 provided comparative energy usage information to utility customers with the intent of reducing their energy consumption behavior. These initiatives used experimental design where control and treatment groups are randomly drawn from the identified program population. The approach provided an equal chance for a sample point to be chosen for the control or treatment group. The customers in the treatment group received feedback reports in a simple one-to-two page format with graphics depicting energy usage compared to a previous period (monthly or bi-monthly) and to neighbors with similar homes. The reports offered simple recommendations on actions customers can take to lower their energy consumption. Also, the reports provided encouragement in a form of simple statements or icons that praises the results in the report if the recipient is performing well relative to similar neighbors.
PG&E, SCE and SDG&E used Opower[1] (an Oracle company) as the provider of their Home Energy Reports (HER) program. PG&E started mailing reports to customers in August 2011; SDG&E started its pilot program in July 2011, and SCE officially launched its Phase 1 pilot in December 2012. All three IOUs continued to provide bi-monthly reports to customers throughout 2015.
The IOUs also continued to offer other behavioral programs similar to the HER program in 2015.
· SCE offered the Home Energy Efficiency Survey (HEES-Enhancement) that provided customers information to help them become familiar with ways to control and reduce energy usage in their homes. The HEES-Enhancement program provided customers information about their energy use in the cover letter of the HEES survey. The report aimed to encourage the customers to return the survey to receive personalized tips on saving energy.
· SDG&E offered the Manage-Act-Save (MAS) program that provided homeowners the ability to monitor and take control of their energy usage and earn points for saving energy. Similar to HER, MAS program offered customers comparative reports via mail or email and personalized tips on how to save energy.
· Marin Clean Energy (MCE) continued to provide the Home Utilities Reports (HURs) pilot program. The HURs program provided comparative usage information similar to the HER program but also encouraged customers to visit MCE website for more customized information for contractors, financing and rebates.
The HER, HEES-Enhancement, HURs and MAS programs used experimental design for allocating households in the treatment and control groups.
1.2 Goals and objectives, including but not directly tied to load impact estimation
The primary purpose of the 2015 behavioral program initiative was to reduce energy consumption by motivating no-cost and low-cost energy conservation actions and self-installation of energy savings measures. The initiative also assessed whether the reports caused customers to participate in other energy efficiency programs as tracked by internal databases. The IOUs objectives for their experiments included the following:
1. What is the energy savings when comparing treatment versus control groups? Is the program experimental design by the IOUs acceptable?
2. What analytical methodologies and data collection should be considered to improve the reliability of this study?
3. How is program targeting affecting the energy savings?
4. What are the peak demand savings attributable to the program?
5. What can we say about persistence of energy savings over time as treatment continues?
6. What can we say about the persistence of energy savings following the cessation of treatment?
2 RESEARCHABLE ISSUES
2.1 PG&E HER Program
PG&E continued to implement its Home Energy Reports (HER) program using Opower. The HER program uses a randomized controlled trial (RCT) experimental design. The pilot started in August 2011 and continued sending bi-monthly reports (and, to households in some experimental conditions, on a less-frequent basis) through December 2015. PG&E contracted with Nexant (formerly known as Freeman, Sullivan, Company or FSC) to facilitate the Opower implementation. Nexant, working with PG&E and Opower, designed the treatment and control samples to enable evaluation, as well as additional research in support of PG&E’s program plans.
PG&E currently now has eight HER waves: Beta Wave (August 2011), Gamma Wave (November 2011), Wave One (February 2012), Wave Two (February 2013), Wave Three (July 2013), Wave Four (April 2014), Wave Five (Oct 2014) and Wave Six (September 2015). Each wave started at different times and was drawn from different populations and received slightly different treatments.
In May 2014, PG&E initiated a persistence study to estimate retention of program savings by terminating treatment to 28,000 randomly selected households from the Gamma Wave. In 2015, PG&E added an additional wave and also added reports delivered via electronic mail (“eHERs”) to subsets of respondents in addition to the print-based reports.
Similar to previous evaluations PG&E will employ Nexant to calculate early program impacts for the 2015 HER program. Nexant will address the following key research questions:
· What are the energy savings due to the program? Are the experimental designs employed by PG&E acceptable?
· If relevant, how is program targeting affecting the energy savings?
· Are there downstream/upstream rebate program savings that could be jointly claimed by both the HER program and the rebate program?
· What are the peak demand savings attributable to the program?
· What can we say about persistence of energy savings over time as treatment continues and energy savings following cessation of treatment?
2.1.1 PG&E Evaluation Approach
In 2015, PG&E added approximately 312,000 more customers to the treatment population bringing a total number of customers treated to 1,996,000. DNV GL will verify the random allocation of the new wave using consumption data and will run the new treatment group as well as the older groups through our independent model (with proper adjustments) to compare to Nexant’s results.
DNV GL established datasets from previous program impact evaluations that included subroutines for compiling, reading in, and cleaning customer billing data and appending PG&E program tracking data. We will rely heavily on re-using these routines for the 2015 impact evaluation.
Consistent with previous evaluations, we will apply a fixed-effect model specification that is a standard for evaluating behavioral programs to estimate kWh and therms savings. For the kW analysis, DNV GL will continue to use the hottest three–day period based on DEER definition when identifying the peak periods. The hottest three-day period (or heat wave) will be identified for each of the climate zones in the IOU service territories. The single heatwave with the preponderance of HER participants in the treatment group will be used as the basis for the demand savings claim for the HER program. Peak demand savings will be estimated using a difference-in-differences approach.
For the persistence study, DNV GL will review random assignment of Gamma’s treatment group to the ‘terminated treatment group’ (those who stopped receiving the report) and ‘continued treatment group’ (those who continued receiving the report). DNV GL will assess savings from these two treatment groups and compare with Nexant’s results to validate persistence in savings due to the suspension of reports.
We will validate Nexant’s calculations of savings jointly caused by the Opower program and other rebate program activities following the approach we used in 2014 evaluation. This analysis will include estimating the following:
n kWh, kW and therms joint savings from downstream programs using a load-shape allocated method
n kWh and kW joint savings from upstream lighting programs leveraging estimates from 2012 PG&E Home Inventory Study, 2014 TRC lighting study, or an updated study if available and DNV GL ULP evaluation report.
2.2 SCE HER AND HEES-Enhancement Programs
SCE HER pilot study (Opower1) was initiated in December 2012 and continued sending bi-monthly reports through 2013. In February 2014, SCE implemented Opower2 that is composed of the unused portion of the eligible population developed for Opower 1. In 2015, SCE added a new HER cohort Opower 3; that is composed of 164,800 treatment and 50,315 control accounts for a total population of 215,115.
SCE hired Applied Energy Group (AEG) to develop the sample for the SCE Opower3 program and to conduct an early impact evaluation. The sample frame (the Account Pool) started with all single-family accounts in SCE’s service territory that met SCE’s program requirements and was then restricted to accounts in Climate Zones 8, 9, 10, and 14. SCE then removed any account included in the Opower1, Opower2, HEES, or Energy Pledge programs. The primary goal of the sample design was to develop a sample representative of the total population of SCE’s residential accounts while also meeting minimum TRC Benefit-Cost requirements. Since smaller accounts tend to save less energy, driving down the overall cost-effectiveness, SCE excluded the lowest quartile of usage. SCE also focused on accounts with email addresses in order to improve the response rate to Opower’s program.
In addition to the HER program, SCE initiated another behavioral program called Home Energy Efficiency Survey Enhancement (HEES-Enhancement) in March 2014 through June 2015. This initiative involves a push marketing campaign that encourages customers to participate in a survey about their energy use in order to receive a personalized report. A total of 200,000 households were randomly assigned to the treatment and control groups for the pilot. SCE hired AEG to develop the sample for the SCE HEES-Enhancement pilot. SCE conducted the early impact evaluation for the 2014 HEES in-house. Although SCE was able to detect statistically significant savings, the measurable savings were less than expected. As such, SCE decided not to extend the pilot beyond June 2015. For this evaluation, DNV GL will estimate pilot results for the full 18 months (2014-2015).
The key research questions for these studies are:
· What are the energy savings due to the HER and HEES-Enhancement programs? Is the program experimental design by the IOUs acceptable?
· Are there downstream/upstream rebate program savings that could be jointly claimed by the HER/HEES-Enhancement programs and other the rebate programs?
· What are the peak demand savings attributable to these programs?
· What can we say about persistence of energy savings following cessation of treatment for Opower1 given that Opower 1 was suspended two years ago?
2.2.1 SCE Evaluation Approach
DNV GL has thoroughly reviewed SCE’s sample design documentation[2] for allocating treatment and control for Opower3 and HEES-Enhancement and found the process undertaken to meet the expectations for an experimental design. In the 2015 evaluation, DNV GL will verify random allocation of the Opower3 wave and HEES-Enhancement program using consumption data.
DNV GL established datasets from Opower1 and Opower2 impact evaluation that included subroutines for compiling, reading in, and cleaning customer billing data and appending SCE program tracking data. We will rely heavily on re-using these routines for the 2015 impact evaluation of HER and HEES-Enhancement program. DNV GL will also review AEG’s datasets and codes to check for any inconsistencies.
DNV GL will apply a fixed-effect model specification that is a standard for evaluating behavioral programs and compare findings to AEG’s results. For the kW analysis, DNV GL will continue to use the hottest three–day period based on DEER definition when identifying the peak periods. The hottest three-day period (or heat wave) will be identified for each of the climate zones in the IOU service territories. The single heatwave with the preponderance of HER participants in the treatment group will be used as the basis for the demand savings claim for the HER program. Peak demand savings will be estimated using a difference-in-differences approach.
DNV GL will also calculate savings jointly caused by the HER/HEES-Enhancement programs and other rebate program activities following the approach used in the 2014 evaluation. This analysis will include estimating the following:
n kWh and kW joint savings from downstream programs using a load-shape allocated method
n kWh and kW joint savings from upstream lighting programs leveraging estimates from 2012 PG&E Home Inventory Study, 2014 TRC lighting study or an updated study if available and DNV GL ULP evaluation report.
2.3 SDG&E HER and Manage-Act-Save Programs
SDG&E started their HER pilot program (Opower-1) using Opower in July 2011 and continued the pilot through 2015. DNV GL participated in the establishment of the RCT experimental design for the SDG&E HER Program by randomly assigning the pre-selected eligible population of approximately 40,000 households into treatment and control groups back in 2011. DNV GL also conducted the impact evaluations of the HER pilot program for the first three years.
In November 2014, SDG&E expanded the HER program (Opower-2) to include 95,002 treatment households that are enrolled in the SDG&E online billing and energy information system (MyAccount). Customers in the new treatment group were assigned to receive either email or paper reports.
In addition to the HER program, SDG&E also offered another behavioral program known as Manage Act Save (MAS) program using Simple Energy. Similar to HER, the MAS program provided customers paper reports or email reports with information on their personal consumption and programs that can help them save energy. The pilot program (MAS-1) started in July 2013 with around 200,000 treatment households but was compromised due to some processing error.