Evaluation
of the
Canada-U.S. Clean Energy Dialogue
February 2011
Report Clearance Steps
Report sent for management response / September 2010
Management response received / January 2011
Report completed / January 2011
Report approved by Departmental Evaluation Committee (DEC) / Month 2011
Acronyms used in the report
ADM / Assistant Deputy MinisterCAA / Clean Air Agenda
CARA / Clean Air Regulatory Agenda
CCS / Carbon capture and storage
CED / Clean Energy Dialogue
CESD / Commissioner of the Environment and Sustainable Development
DFAIT / Foreign Affairs and International Trade Canada
DEC / Departmental Evaluation Committee
DG / Director General
DGMC / Directors General Management Committee
DM / Deputy Minister
DOE / Department of Energy (U.S.)
EC / Environment Canada
FTE / Full-time equivalent
FY / Fiscal year
G&C / Grants and Contributions
GHG / Greenhouse gas
HMARF / Horizontal Management and Accountability Reporting Framework
IPCC / Intergovernmental Panel on Climate Change
MOU / Memorandum of Understanding
NGO / Non-government organization
NRC / National Research Council (Canada)
NRCan / Natural Resources Canada
NRTEE / National Round Table on the Environment and the Economy
R&D / Research and development
RD&D / Research, development and demonstration
S&T / Science and technology
TBS / Treasury Board Secretariat
UNFCCC / United Nations Framework Convention on Climate Change
U.S. / United States (of America)
Acknowledgements
The Evaluation Project Team was led by Linda Lee, under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included Melissa Glen and Edmund Wolfe from Natural Resources Canada, Dorota Geissel and Felicia Minotti from Foreign Affairs and International Trade Canada, as well as Ashley Campbell and Katheryne O’Connor from Environment Canada. The team would like to thank the numerous individuals who provided assistance to the project, including:
· All interviewees who provided their insights on the Canada-U.S. Clean Energy Dialogue; and
· All departmental personnel who responded to our inquiries and requests for documentary evidence, and provided detailed comments crucial to the development of this report and the associated appendices.
This report was prepared for Environment Canada by Marbek Resource Consultants Ltd., in collaboration with Stratos, Inc. and with Alison Kerry.
Table of Contents
Executive Summary i
1.0 Introduction 1
2.0 Background 2
2.1 Profile of the Clean Energy Dialogue 2
2.2 Management Structure 4
2.3 Resource Allocation 7
2.4 Program Logic Model 7
3.0 Evaluation Design 9
3.1 Purpose, Objectives and Scope of the Evaluation 9
3.2 Evaluation Approach and Methodology 10
3.3 Limitations 10
4.0 Findings 11
4.1 Rating of Findings 11
4.2 Relevance 12
4.3 Performance 18
5.0 Conclusions 45
5.1 Issue 1: Continued Need for Program 45
5.2 Issue 2: Alignment with Government Priorities 45
5.3 Issue 3: Alignment with Federal Roles and Responsibilities 46
5.4 Issue 4: Achievement of Outcomes 46
5.5 Issue 5: Efficiency and Economy 47
6.0 Recommendations 48
7.0 MANAGEMENT RESPONSE 50
Annex 1: Evaluation Issues, Questions, Indicators and Data Sources/Methods 56
Annex 2: Summary of Findings Table 61
Annex 3: Outputs Achieved According to CED Performance Measurement Plan Indicators 63
Annex 4: Bibliography 65
List of Tables and Figures
Figure 1: CED Management Structure 6
Figure 2: CAA International Actions Theme Logic Model, amended with the Clean Energy Dialogue 8
Table 1: CED Secretariat and Working Group Funding and FTEs by Department 7
Table 2: Legend – Rating of Findings 11
Table 3: Status of CED Recommended Projects 30
Table 4: CED Budget and Spending for 2009–2010 43
Environment Canada
Audit and Evaluation Branch Evaluation of the Canada-U.S. Clean Energy Dialogue
Executive Summary
Scope and Objectives of the Evaluation
The Canada-U.S. Clean Energy Dialogue (CED) is one of five program components grouped under the International Actions Theme of the Clean Air Agenda (CAA). This evaluation of the CED was conducted to inform senior management and to fulfill Treasury Board Secretariat requirements. Although it is part of the CAA International Actions Theme, the CED was evaluated separately because it was added to the theme later and had separate funding. The specific objectives of the evaluation were to determine the ongoing relevance and performance of the CED. Given the early timing of the evaluation, which occurred before the end of the CED’s second year, the focus was on program outputs and the achievement of early outcomes.
The Canada-U.S. Clean Energy Dialogue
The Canada-U.S. Clean Energy Dialogue was initiated by Prime Minister Harper and President Obama at their meeting in Ottawa in February 2009. As described in the CED Action Plan, “the CED was created to enhance collaboration on the development of clean energy technologies to reduce greenhouse gas emissions and address climate change.” In Canada, the CED is led and coordinated by Environment Canada (EC), with implementation carried out by Natural Resources Canada (NRCan), EC and Foreign Affairs and International Trade Canada (DFAIT). Activities aim to increase collaboration between Canada and the U.S. in three areas: (1) developing and deploying clean energy technologies, with a focus on carbon capture and storage (CCS); (2) building a more efficient electricity grid based on clean and renewable electricity; and (3) expanding clean energy research, development and demonstration (RD&D). The CED has received $7.17 million for a two-year period from 2009–2010 through 2010–2011.
Evaluation Methodology and Findings
The evaluation addressed five key issues: (1) Continued need for the program; (2) Alignment with government priorities; (3) Alignment with federal roles and responsibilities; (4) Achievement of expected outcomes; and (5) Demonstrated efficiency and economy. To examine these issues, and the specific evaluation questions related to each issue, the evaluation employed two main lines of inquiry: (1) document review; and (2) key informant interviews. The evaluation was limited in its ability to comment on longer-term outcomes, as the CED had only been in place for 18 months at the time of data collection. As well, the evaluation was limited in its ability to gain access to U.S. officials and private-sector stakeholders (due to the timing of the exercise and the tight timeframe) and thus limited in its ability to obtain external perspectives on the ongoing relevance and performance of the CED.
Findings and Conclusions
The key findings and conclusions of the five evaluation issues for the CED include:
Issue 1: Continued Need for Program – There is a clear need for the CED in order for Canada and the U.S. to meet international climate change needs related to clean energy technologies.
The CED clearly responds and is connected to international climate change needs, and continued work is required on clean energy technologies to address climate change, particularly in the areas of CCS, energy efficiency and renewable electricity. In addition, since Canada and the U.S. have aligned their climate change targets under the Copenhagen Accord, it is beneficial for them to work together.
Issue 2: Alignment with Government Priorities – The CED is clearly aligned with federal government priorities and the priorities of the participating departments.
The CED has been implemented in response to a direct commitment by Canadian and U.S. leaders and is aligned with a number of federal Throne Speeches. The CED is also aligned with the participating departments’ mandates and priorities.
Issue 3: Alignment with Federal Roles and Responsibilities – There is a clear federal role to deliver on Canada-U.S. engagement; however, provinces also have an important role.
The federal government has the appropriate lead role for the CED, as activities relate to international dialogue to address transboundary/global environmental issues. However, the provinces have responsibility over most forms of energy extraction, production and use, and thus also have an important role to play. While there are other opportunities for cross-border dialogue on climate change, as well as funding for clean energy projects, these initiatives are not duplicative of the CED.
Issue 4: Achievement of Outcomes – The CED has achieved its expected outputs and has demonstrated progress towards achieving its immediate outcome. As well, there is some evidence that the CED is meeting its intermediate outcome of increased collaboration on clean energy RD&D and making progress in building the foundation for compatible policy frameworks. In addition, there is some evidence that Canada’s profile as a provider of climate-friendly technologies has been raised as a result of its commitment to identify and develop clean technologies.
Outreach efforts by DFAIT were not generally seen as directly linked with the CED working groups' specific commitments; rather, they have focused on building relationships with U.S. officials and arranging meetings on behalf of the senior officials.
Two key external factors have influenced the delivery of the CED: (1) challenges in the U.S. regarding the use of private-sector advisors in government-to-government dialogues have led to a reduced role for the private sector compared to initial expectations among some Canadian partners; and (2) provincial governments maintain jurisdiction over energy production and administration of energy resources, limiting the scope of tangible federal contributions in these areas.
The evaluation also demonstrates that it would be useful to re-examine CED Action Plan recommendations to reflect the highest priorities of the Canadian and U.S. working groups, should the CED be renewed.
Issue 5: Efficiency and Economy – The CED is operating efficiently and includes a functional management structure and effective performance reporting, although improvements could be made to the collection of performance data on the progress of CED projects.
There is evidence that the CED provides good value for money, with no alternative delivery means apparent; however, some suggestions for improved efficiency were provided by interviewees. Interviewees from most groups involved in electricity grid initiatives noted the need for more engagement to move CED objectives forward. As well, improving linkages to existing funding mechanisms may help to support CED projects and related outcomes. The CED has generally been implemented as planned, but with some under-spending (21.5%) in the first year and some re-allocation of resources from the Secretariat and private-sector advisor funds to working groups. Roles and responsibilities, particularly those relating to how the external and special advisors would contribute directly to the Action Plan’s recommendations, have evolved from when the CED was first introduced.
The overall management structure—with secretariat support provided by Environment Canada—is operating efficiently and is achieving expected results. The CED Secretariat has been collecting information and reporting on its performance, producing two high-profile Report to Leaders documents as planned. However, improvements are required for the collection of accurate and ongoing performance data on the progress of projects recommended in the Action Plan. Different views were noted on the balance of resources between the Secretariat and the working groups. Ultimately, the allocation should be based on allotted roles and responsibilities, which should be reviewed and/or confirmed if the program is renewed.
While there is acknowledgement among staff members from both EC and NRCan as well as working group members, advisors and other stakeholders that the mandate of the CED could fit within NRCan, there is general agreement among interviewees that the CED is functioning effectively under EC’s direction.
Recommendations
The following recommendations were developed based on the findings and conclusions and are directed to the Assistant Deputy Minister (ADM) of the International Affairs Branch at EC, the ADM of the Science & Policy Integration Branch at NRCan and the ADM of the North America Branch at DFAIT. Should the program be renewed, it is recommended:
1. That EC, as lead and in consultation with NRCan and DFAIT, explore linkages between the CED and existing funding mechanisms to maximize synergies and improve the efficiency and effectiveness of the CED in meeting Action Plan recommendations.
The evaluation demonstrated that there is a continued need for the CED in order to meet key international climate change challenges and to align Canadian and U.S. responses to climate change. However, the CED is not a funding mechanism per se and each CED project must rely on other program funding for significant investments. While progress has been made in most recommended areas, improving the linkages to existing funding mechanisms may further help to move CED initiatives forward. As well, there is an opportunity for the CED both to inform and to be informed by the investment choices of other programs involved in clean energy through these connections. Therefore, it would be beneficial for the CED to explore possible avenues to connect with existing funding mechanisms.
2. That NRCan, as lead and in consultation with EC, consider ways to enhance the level of engagement with provincial partners to advance CED objectives.
It would be beneficial for the CED working groups to assess the appropriate level of engagement with external partners. In particular, given the provinces’ responsibility for electricity generation, it would be advantageous for the Electricity Grid Working Group to re-examine its level of engagement with provincial partners to facilitate more active federal-provincial dialogue. While the Electricity Grid Working Group views the CED conferences as sufficient avenues for domestic consultations, interviewees from most other groups involved in electricity grid initiatives (e.g., provincial representatives, advisors, program staff) noted the need for more engagement. There would be value added, therefore, in the Electricity Grid Working Group clarifying the needs of its members and working to enhance the level of their engagement with provinces. This would facilitate moving CED objectives forward in this focus area.
3. That EC, as lead and in consultation with NRCan and DFAIT as appropriate, clarify, document and communicate the roles and responsibilities of all key players involved.
The evaluation found that the roles and responsibilities for key players in the CED need to be more clearly defined and communicated. For example, there needs to be more clarity around the roles and responsibilities of the CED Secretariat and NRCan’s coordinating function for providing support to the Working Groups. As well, the roles and responsibilities of the working groups themselves need to be clarified and understood by all those involved, particularly with respect to planning and reporting requirements. For some of the key players in the CED, roles and responsibilities have evolved over the course of the first year of implementation. For example, the use of external advisors in implementing the CED has been reduced, and expectations for these private-sector partners need to be reviewed and adjusted accordingly. In addition, the outreach role of the DFAIT Special Advisor shifted as the CED’s focus moved to clean energy technology cooperation.