EVALUATION STRATEGY
2015-2019
INDUSTRY.GOV.AU
Further information
For information on other department initiatives please see the department’s website at: www.industry.gov.au/OCE
For more information or to comment on this publication please contact:
Martine Rodgers
Evaluation Unit
Department of Industry and Science
GPO Box 9839
CANBERRA ACT 2601
Telephone: +61 2 6213 7194
The views expressed in this publication are those of the authors and do not necessarily reflect those of the Australian Government or the Department of Industry and Science.
© Commonwealth of Australia 2015
ISBN: 978-1-925092-67-7 (print)
ISBN: 978-1-925092-68-4 (online)
This work is copyright. Apart from use under Copyright Act 1968, no part may be reproduced or altered by any process without prior written permission from the Australian Government. Requests and inquiries concerning reproduction and rights should be addressed to more information on Office of the Chief Economist research papers please access the department’s website at: www.industry.gov.au/OCE.
Creative Commons Licence
With the exception of the Coat of Arms, this publication is licensed under a Creative Commons Attribution 3.0 Australia Licence.
Creative Commons Attribution 3.0 Australia Licence is a standard form license agreement that allows you to copy, distribute, transmit and adapt this publication provided that you attribute the work. A summary of the licence terms is available from http://creativecommons.org/licenses/by/3.0/au/deed.en. The full licence terms are available from http://creativecommons.org/licenses/by/3.0/au/legalcode.
The Commonwealth’s preference is that you attribute this publication (and any material sourced from it) using the following wording:
Source: Licensed from the Commonwealth of Australia under a Creative Commons Attribution 3.0 Australia Licence. The Commonwealth of Australia does not necessarily endorse the content of this publication.
Contents
Introduction 1
Performance measurement and reporting 2
Impact on evaluation activity 3
What is evaluation? 4
Good evaluation practices – a consistent approach to evaluation 5
Governance 6
Executive Board 6
Steering Committees and Reference Groups 6
Evaluation and Audit—what is the difference? 6
Assurance and Audit Committee 6
Internal Audit role 7
Evaluation Unit role 7
Planning for evaluation across the policy and programme lifecycle 8
Evaluation types 8
Evaluation readiness 9
Prioritising evaluation effort 10
A four-year evaluation plan 10
Alterations to the evaluation plan 10
Scaling evaluation 11
Responsibility for conducting evaluations 11
Evaluation approach 12
Lessons learned — taking advantage of completed evaluations 13
Building capacity and capability 14
Fostering a culture of evaluative thinking 14
Building evaluation capability 14
Supporting guidance material 15
Measures of success 16
Evaluation maturity 16
Reviewing the Evaluation Strategy 18
APPENDIX A – Glossary 19
APPENDIX B – Departmental guidance materials 21
Introduction
This Evaluation Strategy provides a framework to guide the consistent, robust and transparent evaluation and performance measurement of programmes and policies in the department.
Evaluations, reviews and performance monitoring provide assurance that policies and programmes are delivering outcomes as intended, performance is tracked—allowing for correction to occur—and inform future policy and programme design. As Australia is called to adapt to changing economic and policy environments, the evidence gained from evaluations and other forms of performance measurement and assessment support the decision-making of government.
For government, and this department, the continual questioning of how we are performing is a critical part of good performance management and accountability. We need to know:
§ Have we achieved what we set out to do?
§ How are we progressing in achieving the department’s strategic objectives?
§ Could we have done things better?
§ Should we continue to do this or do something else?
Through asking these types of questions we gain an understanding of what works and what doesn’t work and why, what is being done well and what is not, what should be pursued and what should not. This knowledge can improve the design and implementation of effective interventions.
The Public Governance, Performance and Accountability Act 2013 (PGPA Act) establishes a core set of obligations that apply to all Commonwealth entities. The Enhanced Commonwealth Performance Framework brings an increase in external scrutiny, and introduces new requirements for strategic planning, measuring and assessing performance, evaluation and reporting.
Reflecting the department’s response to the PGPA Act, the principles outlined in this Evaluation Strategy (the Strategy) will strengthen evaluation and performance measurement capacity in the department and support building a culture of evaluative thinking, ultimately leading to better resource allocation and decision-making and the evolution of programmes.
This Strategy:
§ Outlines the department’s approach to performance measurement and reporting, according to good evaluation practice
§ Establishes a protocol for policy and programme areas to plan for evaluation across the lifecycle of a programme
§ Introduces a strategic, risk-based, whole-of-department approach to prioritising evaluation effort, and illustrates how evaluations may be scaled based on the value, impact and risk profile of a programme
§ Describes how evaluation findings can be used for better decision-making
§ Describes how the department is building evaluation capability and a culture of continuous improvement
§ Outlines how the department will measure its progress in implementing this Strategy.
This Strategy is not intended to be a complete guide to evaluation and performance measurement. It is supported by a range of internal and external resources including:
§ the department’s comprehensive guidance material and templates for planning and conducting an evaluation
§ the department’s Performance Measurement and Reporting Framework
§ the Department of Finance Enhanced Commonwealth Performance Framework
§ the Australian National Audit Office Best Practice Guide—Successful Implementation of Policy Initiatives.
Performance measurement and reporting
The department’s performance measurement and reporting framework supports the implementation of the Enhanced Commonwealth Performance Framework (the Framework) under the PGPA Act.[1]
The Framework enables Commonwealth entities to devise the necessary links between their performance information and their external reporting. Entities are encouraged to adopt performance measurement methodologies that better assess the results of activities and articulate their performance story. The framework introduces a more transparent and cohesive form of performance reporting related to the activities of an entity in achieving its purpose.
Under the PGPA Act Commonwealth entities will be required to produce and publish annually:
A Corporate Plan, which sets out the purpose of the entity and the method of measuring and assessing the entity’s performance in achieving its purpose.
An Annual Performance Statement, which measures and assesses the entity’s performance in achieving its purpose in the reporting period. This statement is published as part of the Annual Report.
There is a strong emphasis on performance monitoring, evaluation and reporting. Further, the Framework establishes a clear cycle of planning, measuring, evaluation and reporting of results to Parliament, ministers and the public. The revised reporting framework allows flexibility in using various data sources to better assess the results of government programmes. Performance measures are tailored specifically for each programme so that they reflect its design, including its specified objectives, inputs, outputs and outcomes.
Traditional Key Performance Indicators (KPIs) can be complemented with other measures which provide a more meaningful link between public resources used and the results delivered. These may include benchmarking, surveys, peer reviews, and comprehensive evaluations.
Given the department’s high level of complexity, a hierarchy has been adopted to measure and report on performance. Performance measures are identified and reported on at the level of the ‘Activity’, ‘Strategic Objective’ and ‘Entity’, reflecting the outputs, impacts and outcomes of the department’s activities. This approach means that performance monitoring and assessment is undertaken at the appropriate levels, where useful performance measures can be developed and quality data is available.
Figure 1: Reporting hierarchy
The department sets out its vision and four strategic objectives in the Strategic Plan 2015-19.
Impact on evaluation activity
Good performance information will draw on multiple sources that offer different perspectives on the achievement of a programme’s objectives. The performance story of a programme is likely to be best supported through a diverse set of measures.
Evaluations provide a balanced performance story through their incorporation of programme logic tools, and assessment against set criteria.They provide meaningful information and evidence on a component’s aim and purpose in terms of its effectiveness and efficiency and the activities that focussed on that purpose. They provide an opportunity to look beyond performance monitoring and reporting and consider how well the programme is achieving its outcomes.
The department responds to growing demand for evidence-based analyses of policy and programme impacts by applying robust research and analytical methods, both quantitative and qualitative, to determine and isolate what works in industry and science policies and programmes.
What is evaluation?
Evaluation is an essential part of policy development and programme management. The continual questioning of what we are trying to achieve and how we are performing enables us to learn and improve what we do, ensuring that decision-making is informed by the best available evidence.
Policy and programme evaluation involves collecting, analysing, interpreting and communicating information about the performance of government policies and programmes, in order to inform decision-making and support the evolution of programmes.
Evaluation helps to answer questions such as:
§ Is the policy contributing to the intended outcomes or any unintended outcomes?
§ Are there better ways of achieving these outcomes?
§ What has been the impact of the programme?
§ Is the policy still aligned with government priorities, particularly in light of changing circumstances?
§ Should the current programme be expanded, contracted or discontinued?
§ Is there a case to establish new programmes?
§ Can resources be allocated more efficiently by modifying a programme or mix of programmes?[2]
Evaluation is integral to continual improvement. It is a not a one-off, or ‘tick the box’ exercise. Evaluation supports:
EvidencebasedPolicy Development / § evidence-based policy development and decision-making
§ a stronger basis for informing government priorities
§ more efficient resource allocation.
Public
Accountability / § the public accountability requirements of programme sponsors and governments
§ the department’s risk-management processes, helping to encourage greater public trust in government.
Learning / § shared learning to improve policy development and programme design and delivery
§ a culture of organisational learning within the department.
Performance
Reporting / § the analysis and assessment of balanced and meaningful performance information to report on progress in achieving strategic outcomes
§ an enhanced ability to achieve government priorities.
Good evaluation practices – a consistent approach to evaluation
If evaluations are to be valuable to decision-makers across government, consistency in approach and planning are required. Evaluations should be conducted to a standard that ensures the information is credible and evidence-based.
The table below outlines the key principles used to guide evaluation in this department.[3]
Evaluation Principles:1. / Integrated / § Evaluation is core business for the department and is not simply a compliance activity
§ Evaluation planning is undertaken at the New Policy Proposal Stage or early in the design of programmes
§ Evaluation results are communicated widely and inform decision-making and policy development.
2. / Fit for purpose / § The scale of effort and resources allocated to an evaluation are proportional to the value, impact, strategic importance and risk profile of a programme
§ The evaluation method is selected according to the programme lifecycle, feasibility of the method, availability of data and value for money.
3. / Evidence-based / § The department applies robust research and analytical methods to assess impact and outcomes
§ Collectors of administrative data strive to attain baseline measurements and trend data in forms that are relatable to external data sets.
4. / Specific and timely / § Evaluation planning is guided by the timing of critical decisions to ensure sufficient bodies of evidence are available when needed.
5. / Transparent / § All evaluation reports are communicated internally unless there are strong reasons to limit circulation
§ The department will move towards publishing more content externally to strengthen public confidence and support public debate.
6. / Independent / § Evaluation governance bodies have some independence from the responsible policy and programme areas
§ Typically, evaluators will have some independence from the responsible programme and policy areas.
Governance
Executive Board
The Executive Board is responsible for oversight of the department’s evaluation activity, including:
§ Endorsing the department’s annual evaluation plan
§ Agreeing the evaluations to be listed and reported on in public documents, including the Corporate Plan and Annual Performance Statement
§ Observing progress against the annual evaluation plan
§ Noting emerging themes from completed evaluations and audits to help inform future policy and programme design
§ Observing information sharing between the Audit Committee and Evaluation Unit regarding evaluation activity within the department, including the implementation of major recommendations
§ Considering whether to publish evaluation reports or summaries to support greater transparency of the department’s performance and dissemination of learning as noted in the National Commission of Audit.
Steering Committees and Reference Groups
Evaluations conducted by the department are overseen by a Steering Committee or Reference Group. A Steering Committee is more appropriate for larger or more complex evaluations, involving multiple stakeholder groups or layers of government. Steering Committees increase transparency, cross-fertilisation and independence. A Steering Committee gives direction to the team conducting the evaluation and ensures the evaluation is of an acceptable quality and is independent. While it is beneficial to have the knowledge of policy and programme area representatives, there should be a level of independence in the membership of Steering Committees.
A Reference Group differs from a Steering Committee in that it is a sounding board for ideas, but does not formally approve or direct the evaluation. The Chair of a Reference Group should be independent from the policy and programme areas.
Evaluation and Audit—what is the difference?
The roles of evaluators and auditors are quite different but they have similar outputs—both inform performance reporting and address public accountability requirements. There are strategic linkages and synergies between the two functions. When planning an evaluation or audit, and identifying lessons learned to inform policy development and programme delivery, evaluation and audit activity should be considered and coordinated. The responsibilities of the Assurance and Audit Committee and the role of Internal Audit and the Evaluation Unit are outlined below to help identify who does what.