Evaluation of the Great Lakes Action Plan IV
Final Report
June 10, 2010
Environment Canada1
Report Clearance Steps
Planning phase completed / December 2009Report completed / March 31, 2010
Report sent for management response / April 29, 2010
Management response received / May 25, 2010
Report approved by Departmental Evaluation Committee (DEC) / June 10, 2010
Acronyms used in the report
AOC / Areas of ConcernBUI / Beneficial use impairments
CESD / Commissioner of the Environment and Sustainable Development
COA / Canada–Ontario Agreement Respecting the Great LakesBasin Ecosystem
DFO / Fisheries and OceansCanada
ECB / Environmental Conservation Branch
GLAP / Great Lakes Action Plan
GLAP IV / Great Lakes Action Plan for Areas of Concern (Phase IV)
GLBEI / Great LakesBasin Ecosystem Initiative
GLEC / Great Lakes Executive Committee
GLSF / Great Lakes Sustainability Fund
GLWQA / Great Lakes Water Quality Agreement
G&C / Grants and Contributions
IJC / International Joint Commission
MC / Management Committee
MNR / Ministry of Natural Resources (Ontario)
MOE / Ministry of the Environment (Ontario)
MOU / Memorandum of understanding
NGOs / Non-governmental organizations
NWRI / National Water Research Institute
OGDs / Other government departments
O&M / Operations and maintenance
PWGSC / Public Works and Government Services Canada
RAP / Remedial Action Plan
RDG / Regional Director General
TB MAF / Treasury Board Management Accountability Framework
Acknowledgments
The Audit and Evaluation Branch, Evaluation Division Project Team led by William Blois, under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project, particularly members of the Evaluation Committee as well as all interviewees who provided insights and comments crucial to this evaluation. This evaluation was conducted and reported by EKOS Research Associates Limited.
Environment Canada1
Table of Contents
EXECUTIVE SUMMARY
1.0Introduction
1.1Program Profile
1.1.1Program Goals
1.1.2Program Activities
1.1.3Stakeholders and Recipients
1.1.4Governance
1.1.5Resources
1.1.6Program Logic Model
1.1.7Performance Reporting and Evaluations
2.0Evaluation Design
2.1Objectives and Scope
2.2Approach and Methodology
2.2.1Document and Literature Review
2.2.2File Review
2.2.3Key Informant Interviews
2.3Challenges and Limitations
2.4Reporting Note
3.0Findings
3.1Relevance
3.1.1Continued Need
3.1.2Alignment with Federal and Departmental Priorities
3.2Program Performance: Design and Delivery
3.2.1Engagement of Partners
3.2.2Work Planning
3.2.3Roles and Responsibilities
3.2.4Resources
3.2.5Monitoring and Reporting
3.2.6GLSF Program Delivery
3.3Program Performance: Achievement of Program Outcomes
3.3.1Program Outcomes
3.4Program Performance: Efficiency
3.5Performance: Program Economy
3.5.1Cost-Effectiveness
3.5.2Alternative Approaches
4.0Conclusions
4.1Relevance
4.1.1Continued Need for the Program
4.1.2Alignment with Federal and Departmental Priorities
4.1.3Consistency with Federal Roles and Responsibilities
4.2Program Performance: Design and Delivery
4.3Program Performance: Achievement of Program Outcomes
4.4Program Performance: Cost-Efficiency
4.5Program Performance: Cost-Effectiveness
5.0Recommendations
6.0Management Response
ANNEXES (UNDER SEPARATE COVER):
Annex A: GLAP IV Committee Structure
Annex B: Evaluation Issues and Questions
AnnexC: Bibliography
Annex D: Sample and Population of GLSF Files
Annex E: File Review Template
AnnexF: Profile of Agreement Files
Annex G: Descriptive Profile of Sample of Federal Projects
Annex H: Interview Guides
AnnexI: Summary of Findings
Annex J: Status of Beneficial Use Impairments and AOCs
Environment Canada1
EXECUTIVE SUMMARY
Background
In the 2009–2010 fiscal year, Environment Canada’s Audit and Evaluation Branch, Evaluation Division, commissioned an evaluation of the Great Lakes Action Plan IV (GLAP IV). This program was selected for evaluation prior to the end of the program’s terms and conditions at the close of the 2009–2010 fiscal year.
The GLAP for Areas of Concern (AOCs) is the primary vehicle under the Great Lakes Basin EcosystemInitiative(GLBEI) for the federal government, with Environment Canada as the lead department, to act to restore AOCs around the Great Lakes area and fulfill Canadian commitments under the Great Lakes WaterQuality Agreement (GLWQA) with the United States. The current fourth phase[1] of the Action Plan, GLAP IV, was approved in 2005 with a budget of $40 million ($8 million per year over five years) to restore, protect and conserve AOCs around the Great Lakes.
The goals of GLAP IV are:
›to make progress on federal[2] actions in eight AOCs: St. Mary’s River, St.Clair River, Toronto and Region, Hamilton Harbour, Jackfish Bay, DetroitRiver, Niagara River and Port Hope; and
›to complete federal actions in seven AOCs: Thunder Bay, NipigonBay, PeninsulaHarbour, Bay of Quinte, WheatleyHarbour, St. Lawrence River and Spanish River.
To achieve these goals, GLAP IV is based on an ecosystem approach: remedial efforts targeted to AOCs under GLAP IV deal with interrelated environmental and sustainable development issues based on beneficial use impairments (BUIs) identified in these areas. As laid out in the GLWQA, remediation in each AOC is guided by Remedial Action Plans (RAPs).
Program funding is allocated to remedial actions (e.g., sediment remediation, habitat restoration), science (e.g., assessment and monitoring of ecosystem health and status of BUIs), and engagement/governance (e.g., engagement of community members, support for RAP committees). Over the life of the program, approximately 40 science and monitoring projects led by federal partners and over 200 grants and contributions (G&C) projects under the Great Lakes Sustainability Fund (GLSF)were approved for funding.
Evaluation Issues
The evaluation covers GLAP IV program activities from the 2005–2006 to 2009–2010 fiscal years. The evaluation issues include the following:
›Relevance: Is there a continued need for a GLAP? Is GLAP IV aligned to federal government priorities? Is GLAP IV consistent with federal roles and responsibilities?
›Design and Delivery: Is the GLAP IV design appropriate for achieving expected program results?
›Achievement of program outcomes: To what extent have intended outcomes been achieved as a result of GLAP IV? Have there been any unintended (positive or negative) outcomes?
›Efficiency: Is GLAP IV undertaking activities and delivering products in the most efficient manner?
›Economy: Is GLAP IV achieving its intended outcomes in the most economical manner?
Methodology
Data were collected for the evaluation using multiple lines of evidence. These included a document review, a review of 39 GLSF G&C agreement files, a qualitative analysis of 21 federalpartner project files, and a total of 46 key informant interviewswith departmental program managers and federal partners (n=11); representatives of various committees (n=10); federal project proponents (n=4); GLSF project proponents (n=10); unsuccessful applicants (n=4); RAP committee members (n=4); and international or academic experts on aquatic ecosystem remediation (n=3). Key challenges for the study included a reliance on internal sources of evidence and limited availability of program activity information and financial data with respect to GLAP IV.
Evaluation Findings
Relevance
GLAP IV remains a relevant program that enables the federal government to address the continued need for restoration and maintenance of the Great Lakes AOCs. While recent trends in Great Lakes’ ecosystem conditions are variable, historical sources of stress, combined with new challenges, are leading to negative impacts in many areas of the lakes. Environmental need, together with the social and economic benefits of the Great Lakes, supports the continued relevance of the program. Public opinion is consistent with scientific information and key informant views on the environmental and societal importance of the Great Lakes and the continued need for the GLAP.
There is little redundancy risk associated with GLAP IV. The program uniquely addresses federal commitments in the GLWQA and the Canada–Ontario Agreement Respecting the Great Lakes Basin Ecosystem (COA) by providing a framework to address BUIs linked to AOC RAP priority areas. Unlike other public funding (federal, provincial and municipal) and non-public funding (industry, foundations, nongovernmental organizations [NGOs], etc.), GLAP IV is targeted exclusively to AOCs. Key informants express few concerns about the possibility of duplication or overlap between GLAP IV and other programming. Rather, GLAP IV is seen as being highly complementary and as a catalyst in orienting other funding sources to provide support in the AOCs.
GLAP IV is consistent with federal and departmental priorities. The program is part of the GLBEI and is one of six ecosystem initiatives at Environment Canada under the Priority Ecosystem Initiatives Sub-Activity. The GLBEI supports the Department’s Strategic Outcome that “Canada’s natural capital is restored, conserved and enhanced.” GLAP IV outcomes and priority investments also support a number of other sub-activities within Environment Canada’s program activity architecture, including Aquatic Ecosystems, Wildlife, Priority Ecosystems, Ecosystems Sustainability, and Assessment and Ecological Monitoring.
Environment Canada’s GLAP IV is aligned to support federal government commitments and obligations under the GLWQA and COA, as well as federal government priorities outlined in the 2007 Speech from the Throne and two recent federal budgets. Legislative authorities for the GLAP IV include the Canada Water Act, the International Boundary Waters Treaty Act, and the Canadian Environmental Protection Act.
Design and Delivery
Overall, GLAP IV was viewed as a sensible model to achieve intended outcomes. Engagement of partners represents a critical and beneficial aspect of the GLAP IV program delivery, with partnerships occurring at many levels and in many ways. Formal agreements between Canada and the United States (GLWQA) and Canada and Ontario (COA) represent the foundation for commitments of the jurisdictional parties with respect to the AOCs. Implementation involves horizontal partnerships within Environment Canada and across the federal government. RAP committees provide a forum for jurisdictions (federal, provincial, municipal and First Nations) and key stakeholder groups (conservation authorities, community groups and NGOs) to coordinate their efforts through to the delisting process. The funding structure of GLSF projects promotes partner engagement at the local and regional level with partner contributions including cash and in-kind funding, assistance with delivery, and participation in advisory or research capacity.
The implementation of GLAP IV diverged from the original design of the program in several areas. Although the fiveyear workplans required by federal partner departments to access GLAP IV funds were considered to be a positive and worthwhile exercise,the work planning, annual reporting, review and updating processes did not occur as intended, which resulted in a loss of flexibility, responsiveness and accountability.
Due to the horizontal nature of the initiative and broad changes at Environment Canada, GLAP IV governance and management activities do not reflect those outlined in the management framework developed for the program. Areas identified as a potential source of confusion or concern by federal key informants include an inadequate instrument to ensure that partner contributions to program goals are fulfilled, and clarity of roles and responsibilities with respect to delisting AOCs.
For federal partners who received their recommended funding, the amount of GLAP IV funding was seen to be adequate, with the caveat that they would like future funding to be indexed for inflation. However, some federal partners report receiving significantly fewer resources than their original approved funding levels, which they feel impactednegativelyon their ability to implement planned activities. For federal partners that received their GLAP IV funding allocation in full, a more pressing concern was the high ratio of operations and maintenance (O&M) to salary dollars and inability to convert these dollars (i.e., from O&M to salary), which left many groups with insufficient salary dollars to hire scientific and technical staff. GLSF project proponent and committee key informants tended to suggest that the GLAP IV focus and resources be expanded to include priority areas and activities that extend beyond the designated AOCs.
GLAP IV monitoring and reporting activities are undertaken to meet annual reporting obligations associated with both the GLWQA and the COA. At the level of funded projects, GLSF project reporting is quite complete and the majority of reviewed GLSF files provided clear evidence of intended output achievements, however, limited evidence was provided on the outcomes of remedial actions taken in the AOCs. While federal proponents were supportive of increasing reporting requirements for GLAP IV projects to enhance performance measurement and accountability, regular reporting of federal science and monitoring projects is not a formal requirement and reporting to the GLAPIV program is ad hoc.
Project proponent key informants indicate a high level of satisfaction with the GLSF program. Program priorities are considered to be clear and appropriate. The application process is regarded as clear and transparent by most funding recipients and selection criteria are viewed to be explicit and well communicated. Overall, the GLSF proposal review process is perceived to be logical and GLSF funding decisions are viewed as strategic and fair. Program staff are seen by project proponents to be accessible and supportive. Modifications to the GLSF program (including a shift from the use of memoranda of understanding to contribution agreements), implemented by Environment Canada in 2007, have challenged the timely delivery of the program, reduced flexibility and increased uncertainty among partners. Project proponents indicate a preference for streamlining the application and approval process, for multi-year agreements and for more straightforward funding agreements.
Achievement of Program Outcomes
In general, key informants had favourable impressions on the performance of the program with respect to the achievement of immediate and intermediate outcomes, a finding that is supported by the program documentation and review of GLSF and federal project files. As indicated, engagement of partners at the local level and across jurisdictions and scientific communities is perceived to be a strength of the program. This is supported by GLAP IV funds allocated to federal partners to facilitate coordination and management of GLAP IV (e.g., with the provinces and First Nations). Engagement of partners and participation at the local level is facilitated through committees, informal networks and the RAP structure (which also receive GLSF funding support). With respect to remedial actions, addressing pollution (through sediment remediation, support to municipal infrastructure improvements and reducing non-point sources of pollution) is a significant priority for the program and is allocated almost onehalf of the GLAP IV dollars. While the success of federal partners’ efforts is difficult to discern due to gaps in the federal project files, GLSF projects that supported this objective were found to be welldocumented in terms of outputs and experienced few challenges in implementation. BUI assessment and monitoring work is occurring on many fronts to assess the status of BUIs, effectiveness of restoration activities and defining BUI goals/targets. Fewer federal projects were funded for habitat restoration (though this is a particular focus of GLSF). Habitat projects are more easily documented in terms of outputs (e.g., plantings, wetland acreage), but these projects are also more subject to implementation challenges. Overall, program investments advance activity in each AOC to some degree (though some coordination projects target all AOCs generically). Note that while group 1 AOCs (those closest to delisting) were intended to be assigned a higher priority for GLAP IV investments, the number of projects funded in these group 1 AOCs is lower than in the group 2 AOCs.
Achievement of the program’s longer-term outcomes is much less evident, and the majority of BUIs that were originally identified in AOCs still exist. Of the over 100 BUIs identified, only 20 percent have been restored. In consideration of the status of the BUIs, the program has identified outstanding priority actions for each AOC and projected timelines for delisting. Only two AOCs have been delisted and one AOC designated an area in recovery and these occurred prior to GLAP IV. The original program goal of completing federal actions in seven group 1 AOCs has not been achieved. Three AOCs are expected to be delisted or designated as Areas in Recovery within the next one to two years, and a fourth, WheatleyHarbour,has just been delisted (announced on April16, 2010). With respect to the remaining AOCs, the picture is much more complex and the timeframes for delisting are longer-term, with most of these AOCs estimating delisting as occurring between 2015 and 2020. For committee and expert key informants, the overall importance of the contribution made by GLAP IV to the restoration and maintenance of AOCs is viewed as being very significant, if not critical.
For key informants, positive external factors that support program success include: leveraged funding, particularly infrastructure funding; United Statesinvestments; and community/political engagement (e.g., the Great Lakes and St. Lawrence Cities Initiative). External factors that interviewees identified as having the potential to negatively affect the success of GLAP IV include: new and changing ecosystem issues; and the economic downturn. Unintended outcomes of GLAP IV cited by key informants tend to be positive and focus on unexpected interest and engagement of non-targeted groups (e.g., landowners) and the general public, as well as on unanticipated opportunities for collaboration and knowledge transfer.
Efficiency
The cost-efficiency of GLAP IV is difficult to determine with existing information due to the unbundling[3] of GLBEI funds and activities and the introduction of a shared, resultsbased accountability approach to environmental initiatives. The impact on financial data availability resulting from the changes implemented to Environment Canada’s resource allocation and accountability structure is that GLAP IV funds are not specifically coded at the program level and the A-base support allocated to GLAP IV is not known.
Financial analysis of the GLSF program component indicates that for every contribution dollar, $0.24 is spent on program administration (salary and O&M). This amount compares favourably to Environment Canada’s EcoAction Community Funding Program, which has a cost-efficiency ratio of $0.39, but unfavourably with the Habitat Stewardship Program ($0.08) and the Invasive Alien Species Partnership Program ($0.13). Like EcoAction, the delivery of the GLSF involves additional staff responsibilities such as participation in RAP committees.
For key informants, there is an impression of efficient program delivery both overall and at the project level, with a number of factors identified that support costefficiency at the program and project levels. Some suggestions to improve efficiency were nonetheless received.