Audit and Evaluation Branch Evaluation of…

Evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE)

July 2011

Environment Canada1

Audit and Evaluation BranchEvaluation of the PPRE

Report Clearance Steps

Planning phase completed / February 2011
Lessons learned completed / April 2011
Report completed / April 2011
Report approved by Departmental Evaluation Committee / July 2011

Acronyms used in the Report

CACsCriteria air contaminants

CAPClean Air Partnership

CARAClean Air Regulatory Agenda

CEPACanadian Environmental Protection Act

FCMFederation of Canadian Municipalities

FCOFleet Challenge Ontario

FTEFull-time equivalent

G&CGrants and contributions

GHGsGreenhouse gases

HDDHeavy-duty diesel

MSCMy Sustainable Canada

NCDCNational Clean Diesel Campaign

NGONon-governmental organization

NOxNitrogen oxides

NRCanNatural Resources Canada

O&MOperations and maintenance

PAAProgram Activity Architecture

PMParticulate matter

PPREPilot Project on Reducing Emissions from Vehicles and Engines

SPPSecurity and Prosperity Partnership of North America

U.S. EPAUnited States Environmental Protection Agency

Acknowledgements

The Evaluation Consultant Team of Stratos, Inc. and Alison Kerry would like to thank Nicola Scahill and Manjit Kerr-Upal for their ongoing input, provision of documentation, and review of key deliverables during the evaluation process.They would also like to thank the interviewees who provided their insights on the pilot project.

Input from the federal government’s Evaluation Division was provided by Robert Tkaczyk under the direction of the Environment Canada Evaluation Director, William Blois.

The evaluation was conducted by Stratos, Inc. in association with Alison Kerry.

Table of Contents

Executive Summary

1.Introduction

2.Background

2.1Profile

2.1.1Objectives and Expected Outcomes

2.1.2Stakeholders and Beneficiaries

2.2Governance

3.Evaluation Design

3.1Purpose and scope

3.2Evaluation Issues and Questions

3.3Evaluation Approach and Methodology

3.4Limitations

4.Findings by Evaluation Issue

4.1Rating of Findings

4.2Relevance – Continued Need for the Program

4.3Relevance – Alignment with Federal government Priorities

4.4Relevance – Consistency with Federal Roles

4.5Performance – Design and Delivery – Performance Information

4.6Performance – Design and Delivery – Program Design

4.7Performance – Design and Delivery – Barriers

4.8Performance – Design and Delivery – Communications

4.9Performance – Design and Delivery – Best Practices

4.10Performance – Effectiveness – Outputs and Outcomes

4.11Performance – Effectiveness – Intended Outcomes

4.12Performance – Efficiency

4.13Performance – Economy

5.Conclusions

6.Lessons Learned

Annex1. Bibliography of Documents Reviewed

Annex 2: Literature Review

Annex 3. Interviewees and Interview Guides

Annex 4. Summary of Findings (Table)

Audit and Evaluation BranchEvaluation of the PPRE

Executive Summary

Scope and Objectives of the Evaluation

A formal, independent evaluation is required to assess the relevance and performance of the Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE). The scope of the evaluation is to cover the three years of the program and all related project activities. This requirement fulfills the commitment made in the original program design for the PPRE. As per the requirements presented in the Directive on the Evaluation Function, this evaluation will address value for money by including clear and valid conclusions about the relevance and performance of the pilot program. As the funding is not ongoing, and there is no decision required on program renewal, recommendations are not part of this evaluation. Rather, the focus is on lessons learned that may inform related future work and regulatory development.

Pilot Project on Reducing Emissions from Vehicles and Engines

The PPRE was an Environment Canada initiative, under the Security and Prosperity Partnership of North America (SPP), funded at the level of $1.2million over three years (2008-11). The objectives of this pilot project were for Environment Canada and the United States Environmental Protection Agency (U.S. EPA)to collaborate on initiatives aimed at reducing emissions from vehicles and engines in order to better utilize the capabilities of each agency, and to share information on innovative programs such as retrofitting diesel engines for lower emissions. Funding was required to work on collaborative initiatives with the U.S. EPA, and to work with stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Evaluation Methodology and Design

The evaluation addressed five key issues: (1) Continued need for the program; (2) Alignment with government priorities; (3) Alignment with federal roles and responsibilities; (4) Achievement of expected outcomes; and (5) Demonstrated efficiency and economy. To examine these issues, and the specific evaluation questions related to each issue, the evaluation employed three lines of inquiry: (1) document review; (2) literature review and comparison of similar programs; and (3) key informant interviews. The evaluation scope focused on Environment Canada’s pilot project to work with stakeholders (within Canada) to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Evaluation Conclusions

The key findings from the evaluation, detailed in the report, are summarized below.

Relevance

  1. The pilot project and its respective activities are relevant and continue to be needed to address heavy-duty diesel (HDD) greenhouse gas emissions and criteria air contaminants produced by public fleets.
  2. The pilot project is directly aligned with federal government priorities and the strategic outcomes of Environment Canada to take action on climate change and reduce air emissions.
  3. The activities of the pilot program are aligned with federal roles and responsibilities.

Performance – Project Design and Delivery

  1. Performance information is provided by funded partners, primarily at the output level, in a timely and relevant manner. Data pertaining to the project’s performance indicator are currently being collected and will be reviewed at the end of the pilot. The evaluation did not find evidence of a formal internal reporting mechanism to senior managers or federal partners, though information was shared throughout the project on an informal basis.
  2. The project was designed appropriately and included a clear and plausible link between program activities, outputs and expected results. The resources allocated to this project were commensurate with the expected results.
  3. The pilot project identified that access to funding and the ability to work with the variety of players who are involved with municipal fleets (e.g. ranging from municipal governments to subcontractors and third parties) may be barriers to implementing a similar project on a national scale.
  4. On behalf of Environment Canada, project partners engaged municipal fleet operators regularly and communicated the outputs from project activities in a timely and relevant manner. Correspondence and communication within the federal community was informal and in some cases inconsistent.
  5. The pilot project was successful in developing a best practices guide and a resources guide for municipal fleet managers, and in providing advice on how to move forward with addressing emissions from school buses.

Performance – Effectiveness

  1. The pilot project successfully generated a number of important outputs, including a best practices guide and a resources guide, fact sheets, and multiple reports that increased the level of knowledge about and number of approaches to addressing the on-road HDD municipal fleet and school bus sectors. One planned output (a baseline and profile of municipal fleets in Canada) was not achieved. Data are not yet available toassess the achievement of the project’s expected outcome.
  2. The evaluation did not find any negative unintended outcomes.
  3. Environment Canada successfully partnered with organizations that were experts in the subject matter and/or maintained a comprehensive understanding of the key organizations and individuals within the municipal fleet sector. In one case, the Department successfully leveraged funding from an organization to support the project.
  4. The evaluation found that the project is on track to achieving its intended outcome in an economical manner.

Lessons Learned

The evaluation identified a number of lessons learned, and suggestions on how to improve the delivery of similar projects complementary to regulatory development. These are detailed in Section6, and are summarized below.

Key Lessons Learned from the PPRE:

  • Allocating an appropriate amount of time to designing a program/project can lead to relevant and valuable results. Environment Canada allocated one year to design a successful pilot project that was relevant and met a demonstrable need that was not duplicative of other federal efforts.
  • Partnering with organizations that maintain expertise and existing networks can lead to effective results. Environment Canada worked with non-governmental organizations (NGOs) that maintained existing networks and a comprehensive understanding of municipal fleets across Canada. This approach enhanced program efficiencies by creating immediate access to existing networks, and provided greater assurance that project deliverables were credible and were useful to the municipal fleet community.
  • When provided with the appropriate tools and funding, Canadian municipalities can engage in actions to reduce emissions from their fleet vehicles. The PPRE has shown that by building the capacity of municipalities, providing tools, and providing assistance in accessing funds, emission reductions can be supported and achieved.

Suggestions for Future Projects:

  • A one-window approach, providing information and resources to reduce emissions from all types of vehicles in the municipal fleet, would be beneficial. There is merit in addressing the emissions from all vehicle types that are present in the municipal fleet more broadly, rather than just focusing on HDD. Fleet managers deal with multiple types of vehicles and make integrated decisions that can best be informed and supported by a more comprehensive approach to addressing emissions from all vehicle types.
  • Long-term funding is required to ensure action is taken to reduce fleet emissions. The pilot project has increased the general level of awareness of municipal fleet managers and has provided a number of resources to reduce the emissions associated with municipal fleets. However, the pilot program has also underlined a need for longterm funding to implement solutions to reduce emissions produced by municipal fleets. This pilot project has demonstrated that long-term funding from sources with municipal jurisdiction should be identified.
  • The Government of Canada can play an important role in identifying innovative technologies for reducing vehicle emissions. The Government of Canada can play an important role in testing of technologies for reducing on-road vehicle emissions. The PPREfunded project in Ontario led to the identification of retrofit technology that has a proven ability to reduce emissions produced by school buses.
  • Environment Canada is best placed as an enabler for reducing vehicle emissions. The evaluation highlighted that this pilot project was a unique opportunity for Environment Canada to lead a pilot project that identified a gap, was complementary to HDD regulations, and addressed greenhouse gases and air pollutants. However, other federal departments, such as Natural Resources Canada, are now better placed to lead comprehensive capacity-building programs in this area, because of their mandate and emerging work under the FleetSmart program.
  • A national emissions baseline for municipal fleets is required. The pilot project identified the need for a national emissions baseline for municipal fleets, but found it challenging to complete. In order to target cost-effective emission reduction activities and to measure change with municipal fleets, a baseline is required.
  • The impact of a capacitybuilding/outreach program is maximized when delivered jointly with a suite of like-minded programs, including regulations/standards. Although education and capacity building are important tools to help municipalities take action to reduce their emissions, a suite of complementary and reinforcing initiatives, including regulations, provides them with the push and pull required for taking substantive action within their jurisdictions.

Environment Canada1

Audit and Evaluation BranchEvaluation of the PPRE

1.Introduction

The evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE) was undertaken from January to March 2011, in response to a commitment to conduct an evaluation of the original program design documentation submitted to Treasury Board Secretariat officials. The intent of this evaluation is to inform management in Environment Canada, as well as the Treasury Board Secretariat, about the success of the pilot project and lessons learned that may inform related future work.

For the PPRE evaluation, program staff in the Strategic Transportation and Policy and Analysis Sectionof Environment Canada’s Transportation Division, along with a representative from the Department’s Evaluation Division, led the evaluation and performed an oversight role that involved providing input and feedback to the consultant hired to conduct the independent evaluation; approved all deliverables outlined in the contract; and coordinated the management response to the evaluation report.

This report summarizes the evaluation process and findings, conclusions and lessons learned. The report is organized into five sections:

  • Section 2 describes the pilot project, including the governance structure, allocated resources and expected outcomes
  • Section 3 indicates the evaluation objectives, scope, issues and approach taken
  • Section 4 documents the evaluation findings
  • Section 5 presents the evaluation conclusions
  • Section 6 outlines the lessons learned

The report also includes a number of annexes that form the information base for the evaluation’s key findings and conclusions, including a list of reference material, literature review details, interviewees, and a summary findings table.

2.Background

The SPP is an Industry Canada–led initiative that provides a flexible means for dialogue, priority setting, collaboration and action on issues affecting the security, prosperity and quality of life of Canadians, Americans and Mexicans. It addresses diverse issues, such as border facilitation, the environment, food and product safety, and includes measures to improve overall North American competitiveness.

A total of $39.2million was allocated to the SPP over three years (2008-09, 2009-10, 2010-11). Of this allocation, $6.2million was provided to Environment Canada for three projects:

  1. Chemicals assessment ($4million)
  2. A North American air emissions inventory ($1million)
  3. The PPRE ($1.2million)

An overview of the funding provided to Environment Canada can be viewed in Table 1 below.

Table 1: SPP Funding to Environment Canada

Initiative / 2008-09 / 2009-10 / 2010-11 / Total
Chemicals Assessment / $1,190,000 / $1,390,000 / $1,420,000 / $4M
N.A. Emissions Inventory / $163,000 / $418,000 / $419,000 / $1M
PPRE / $120,000 / $540,000 / $540,000 / $1.2M
TOTAL / $1,473,000 / $2,348,000 / $2,379,000 / $6.2M

This evaluation relates solely to the PPRE.

Table 2 summarizes the resources allocated to the Transportation Division for PPRE activities.

Table 2: Resources allocated to the Transportation Division for PPRE activities

Funded Activities / Fiscal Year / Total
2008-09 / 2009-10 / 2010-11
FTE / 0.5 / 1 / 1
Salaries / $50,549 / $107,764 / $107,764 / $266,077
Employee Benefits / $10,109 / $21,553 / $21,553 / $53,215
O&M / $52,770 / $73,009 / $73,009 / $198,788
G&C / $323,664 / $323,664 / $647,328
PWGSC Accommodation / $6,572 / $14,010 / $14,010 / $34,592
Total / $120,000 / $540,000 / $540,000 / $1.2M

2.1Profile

The PPRE complements the Clean Air Regulatory Agenda (CARA) by addressing emissions from existing vehicles and engines that could remain on Canada’s roads for up to 20 years.[1]It targets on-road heavyduty diesel (HDD) vehicles such as trucks and school buses, and explores how best to voluntarily reduce their emissions (greenhouse gases [GHGs] and criteria air contaminants [CACs]). Emissions from these pre-2007 diesel engines contribute significantly to local, regional and global air pollution. The Government of Canada carries out various emission reduction programs targeting HDD vehicles (mainly Class 8 freight vehicles), such as Transport Canada’s ecoFREIGHT program and Natural Resources Canada’s (NRCan’s) FleetSmart program. This initiative complements these programs by focusing on Canada’s existing on-road publicHDD vehicles, such as municipal fleet vehicles.

2.1.1Objectives and Expected Outcomes

The objective of the PPRE, as noted in its program design, is as follows: “Environment Canada and the United States Environmental Protection Agency (U.S. EPA) will collaborate on initiatives to reduce emissions from vehicles and engines to better utilize the capabilities of each agency, as well as share information on innovative programs such as retrofitting diesel engines for lower emissions. Funding is required to work on collaborative initiatives with the U.S. EPA, and to work with stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.”

The first year of the SPP pilot (2008-09) was dedicated to developing the program design, which included consultations with Transport Canada, NRCan and the U.S. EPA to gain knowledge and advice on how the pilot project should be designed. These consultations allowed Environment Canada to better understand what work was already being carried out in this area (e.g., ecoFREIGHT and FleetSmart) and gaps or opportunities that could be addressed through the pilot project. Based on this design phase, two initiatives emerged:

Reducing Emissions from Canada’s On-Road HeavyDuty Diesel Municipal Fleet

The objective of this project was to increase knowledge, awareness and supporting tools for municipalities to reduce their emissions of GHGs and CACs, by highlighting emission reduction opportunities and identifying barriers and challenges to be overcome during the implementation of emission reduction strategies for fleet vehicles. To realize this objective, Environment Canada signed a Contribution Agreement with the Federation of Canadian Municipalities (FCM) in January of 2010, in order to enable the following activities:

  • Establish partnerships with municipal fleet managers’ networks, federal and provincial agencies, environmental and non-governmental organizations (NGOs) and private sector organizations.
  • Create a baseline of the municipal heavyduty fleet within Canada, including a fleet profile, an overview of best practices, and identification of barriers and challenges for initiatives to green fleets.
  • Identify and create an online resource in both official languages for municipal fleet managers to access information on best practices available to achieve emission reductions.
  • Promote in both official languages the online resource and other products and services (e.g., publications) to reduce HDD emissions.
  • Deliver education and training workshops in collaboration with partners to increase the capacity of municipal fleet managers to make decisions about reducing HDD emissions within their fleet profile.
  • Develop a business case for emission reduction initiatives in heavyduty fleets, to be included in a resources guide for fleet managers.

In addition to the work completed with the FCM, Bronson Consulting was contracted to conduct a scoping study of Canadian school bus fleet characteristics, in order to identifykey considerations in the development of any clean air programs for school bus fleets.