FREE STATE DEPARTMENT OF HEALTH
PERFORMANCE MONITORING AND EVALUATION IMPLEMENTATION FRAMEWORK
SEPTBEMBER 2014
TABLE OF CONTENTS
NUMBER / CONTENT / PAGE(i) / List of acronyms and abbreviations / 5
(ii) / Executive Summary / 7
(iii) / Foreword / 8
(iv) / Introduction by the Accounting Officer / 7
(v) / Vision, Mission, Values and Goals / 9
CHAPTER 1 /
UNDERSTANDING MONITORING AND EVALUATION (M&E)
1.1 /Introduction
/ 101.2 /
Legal background
/ 101.3 / Audit of Performance Information / 12
1.4 / Target Group / 13
1.5 / Alignment with other government systems / 13
1.6 /
Monitoring and Evaluation
/ 141.6.1 /
Monitoring
/ 141.6.1.1 /
Guiding principles of Monitoring
/ 141.6.1.2 / Types of Monitoring / 15
1.6.2 /
Evaluation
/ 171.6.2.1 /
Guiding principles of Evaluation
/ 171.6.2.2 /
Types of Evaluation
/ 191.6.2.3 /
Correlation between Monitoring and Evaluation
/ 231.6.2.4 / Why Evaluate / 24
1.6.2.5 / The objectives of Monitoring and Evaluation / 24
1.7 /
The logic of M&E
/ 25CHAPTER 2 /
INSTITUTIONALISATION OF HEALTH MONITORING AND EVALUATION SYSTEM (M&E
2.1 /Presidential Performance Monitoring and Evaluation: Improving Government performance processes
/ 262.2 / The role of the Department of the Premier: PME / 26
2.3 /
Support Structures to strengthen Departmental PME Systems
/ 272.4 /
Governance Structures
/ 312.5 /
Public Sector institutions
/ 322.6 /
Other Institutions
/ 332.7 /
Accreditation Bodies
/ 332.9 /
Training and Research Institutions
/ 332.10 /
FSDOH Research Committee
/ 342.11 / Creation and maintenance of a Performance Information repository creates as: // FSH-VIBE PME system. / 34
2.12 /
Selecting External Evaluators
/ 352.13 /
Ensuring Compliance with Evaluation Standards
/ 352.14 /
Oversight Structures
/ 352.15 /
Financial Resources for effective PME implementation
/ 36CHAPTER 3 /
DESIGNING MONITORING AND EVALUATION SYSTEM
3.1 /Conducting a readiness assessment
/ 363.2 / Choosing outcomes to monitor and evaluate / 36
3.3 / Selecting key indicators to monitor outcomes / 37
3.4 / Baseline data on indicators / 37
3.5 / Planning for improvement – selecting targets / 39
3.6 / Monitoring of the results / 39
3.7 / Evaluation of the results / 39
3.8 / Using the findings / 40
3.9 / Sustaining the M&E system / 40
3.10 / M&E capacity building / 41
3.11 / Change management / 42
CHAPTER 4 / PERFORMANCE INFORMATION REPORTING GUIDELINES
4.1 / Purpose / 42
4.2 / Background of performance information reporting / 42
4.3 / The process of performance reporting / 42
4.3.1 / Performance Reporting format / 43
4.3.2 / Performance Reporting Period (Insync with the budget cycle) / 45
4.3.3 / Verification, validation of reported information / 45
4.4 / Early warning system / 46
4.5 / Improvement Plans and Corrective Action / 46
CHAPTER 5 /
IMPLEMENTATION PROCESS OF THE M&E FRAMEWORK
5.1 /Developmental Phase
/ 475.2 /
Engagement with various stakeholders
/ 475.3 /
Implementation of M&E Framework
/ 475.4 /
Utilisation of M&E results
/ 485.5 /
Ownership of the system
/ 485.6 /
Revision
/ 486 /
CONCLUSION
/ 497 /
LIST OF SOURCES
/ 508 /
LIST OF APPENDICES
/ 51(a)LIST OF ACRONYMS AND ABBREVIATIONS
AGSA Auditor General of South Africa
AoPO Audit of Predetermined Objectives
APP Annual Performance Plan
AR Annual Report
CFO Chief Financial Officer
CHC Community Health Centre
CEO Chief Executive Officer
DIODistrict Information Officer
DPME Department of Performance Monitoring and Evaluation
DHPDistrict Health Plan
DHERDistrict Health Expenditure Review
DHIMSDistrict Health Information Management System
DPSA Department of Public Service and Administration
EMSEmergency Medical Services
FSDOHFree State Department of Health
FMPPIFramework for Managing Programme Performance Information
GMGeneral Manager
GWM&ES Government-wide M&E System
HODHead of Department
IDPInfrastructure Development Plan
IOInformation Officer
LAMLocal Area Manager
M&EMonitoring and Evaluation
MECMember of Executive Council
MPAT Management Performance Assessment Tool
NEPF National Evaluation Policy Framework
NSDA Negotiated Service Delivery Agreement
OBPOutcome Based Priorities
OPSC Office of the Public Service Commission
PFMA Public Finance Management Act
PHCPrimary Health Care
PIPerformance Information
PIPPerformance Information Plan
PMEPerformance Monitoring and Evaluation
POAProgram of Action
PSCPublic Service Commission
QPRS Quarterly Performance Review System
RSARepublic of South Africa
SAI Supreme Audit Institution
SAMEA South African M&E Association
SEDS Social, Economic and Demographic Statistics
SASQAF South African Statistics Agency Quality Assurance Framework
SONAState of the Nation Address
SOP Standard Operating Procedures
SOPAState of the Province Address
STPService Transformation Plan
TORs Terms of reference
(b)Executive Summary
The FSDOH Implementation Framework describe the approach the Department will follow to create and operate an effective PM&E System that will produce credible, accurate evidence based performance information on an ongoing basis that are used to improve service delivery and governance.
The intention of the Framework is to manage government performance, not only through the management ofbudgets and ensuring effective administrative practices, but also by making the results of programmes transparent.
The Policy Framework for the Government Wide Monitoring and Evaluation (GWM&E)
System, published in 2007 by the Presidency, emphasise the importance of monitoring and evaluation in realising a more effective government.The FSDOH adopted the abovementioned document as the PME system to be used.
This enhances the responsibility of managers to develop and implement an integrated monitoring, reporting and evaluation system for the Free State Department of Health.
The system will serve to assess individuals and collectives of people, to evaluate an
institution’s effectiveness and validity of a policy.
For the PME Framework to be effective, accountability must be affected at all levels of management within the FSDOH. Relevant structures and systems have already been established and empowered to support the implementation of the PME policy and systems. Structures with roles and responsibilities at each level of governance will be instituted to form the teams for the implementation of PME.
Each institution will have to decide on the appropriate positioning of the responsibility to manage performance information. Ideally, this capacity should be aligned to the planning and financial management functions. This focus on the overall design and management of indicators, data collection, collation and verification processes within the institution. It must be emphasized that line managers remain responsible for establishing and running performance information systems within their sections, and for using performance information to make decisions. All Units / Sub sections / Divisions will interact with the Performance Monitoring and Evaluation Corporate Team on a continuous basis.
The processes in the Framework ensures that we translate our mandate into a very clear set of outcomes and a few crucial output measures that will help us deliver.
(iii)Introduction by the Accounting Officer
The underlying purpose of the FSDOH PME Implementation Framework is to institutionalize PME practices, improving policy programme performance accountability, decision-making and increasing knowledge about what works.
The Free State Department of Health (FSDOH) has adopted the Government Wide
Monitoring and Evaluation System as well as the National Evaluation Policy Framework. The development of the FSDOH Performance Monitoring and Evaluation (PME) Implementation Framework provide guidelines for the practical implementation of Performance Monitoring and Evaluation practices in the department. The structures developed for the decentralization of PME provides the platform for implementation as well as taking ownership for the PME system. Performance Monitoring and Evaluation will be implemented following the Outcomes Based Approach and will link with various other government assessment systems, e.g. MPAT.
The monitoring and evaluation capacity in the department of Health is limited, therefore developing a culture of compliance. Efforts has been made to establish the conditions to enable a system which would support implementation of the framework and use evaluation results. This includes establishing guidelines, evaluation standards, evaluations competency levels for programme managers, M&E staff and evaluations, training, and quality assessment of evaluations.
The FSDOH PME Implementation Framework takes into account all the relevant policies, legislation and other mandates for which DPME is responsible. The FSDOH PME Strategy accurately reflects the strategic process to be implemented.
If we achieve progress in priority areas identified utilizing this PME Implementation Framework, we will have successfully moved the FSDOH onto a new development path.
Dr. D. Motau
HEAD: HEALTH
September 2014
VISION
Increasing life expectancy through health system effectiveness, driving system change and ensuring sustainable quality services
MISSION
By creating a value driven institution that inculcates operational efficiency and accountability in delivering desired outcomes effectively.
VALLUES
The key determinants of relationships within the Free State Department of Health are:
•Accountability;
•Responsiveness;
•Batho Pele Principles
•Commitment; and
•Integrity.
STRATEGIC GOALS
Goal 1: Provision of strategic leadership and creation of a social compact for better health
outcomes.
Goal 2: Manage the financial affairs for sustainable health service delivery.
Goal 3: Build a strategic and dedicated workforce that is responsive to service demands.
Goal 4: Re-engineer Primary health care to create access to quality services.
Goal 5: Develop, operate and manage infrastructure for compliance and better health
outcomes.
Goal 6:Strengthen information and knowledge management system to optimise
performance and research capability.
Goal 7: Optimise and support implementation of key priority programmes
CHAPTER1: UNDERSTANDING MONITORING AND EVALUATION (M&E)
1.1Introduction
This document presents the Department of Health’s framework for the development and implementation of a Monitoring and Evaluation system. Gathering, analyzing and using information is important to any health programme. M&E is a planning tool that is used to review progress made, the challenges experienced as well as to take corrective measures where necessary to assess the value of what is being done.
The Free State Department of Health has adopted the National Performance Monitoring and Evaluation Policy as well as the Government Wide Monitoring and Evaluation System(2007). The Department of Health shall implement the Government Wide Monitoring and Evaluation system (GWM&ES) focusing on the outcome based approach.
1.2Legal background
The intention to manage government performance, not only through the management of budgets and ensuring effective administrative practices, but also by making the results of programmes transparent, is signalled in the Constitution. Institutions have a responsibility to publish administrative and performance information to account to Parliament and provincial legislatures in accordance with Sections 92 and 114 of the Constitution and to be transparent and accountable to the public in accordance with section 195 of the Constitution.
Accounting officers are responsible for targeting performance and managing PI. In terms of the PFMA Section 27(4), accounting officers of national departments must submit measurable objectives with their draft budgets to Parliament and provincial accounting officers submit to provincial legislatures. In terms of Section 40(1) and (3) accounting officers must provide information on departments’ achievements against their predetermined objectives in the Annual Report; and in terms of Section 55(1) and (3) accounting authorities of public entities should do the same. Furthermore, in Section 38(1)(b) accounting officers of departments and constitutional institutions are responsible for the transparent, effective, efficient, and economical use of resources of the department or constitutional institution.
In terms of the Public Service Act (1994) Section 7A(4)(c) executive authorities determine the reporting requirements of the heads of government components, including public entities, to the head of the principal department to enable oversight of the component in respect of policy implementation, performance, integrated planning, budgeting and service delivery.
The Policy Framework for the Government Wide Monitoring and Evaluation (GWM&E)
System, published in 2007 by the Presidency, emphasised the importance of monitoring and evaluation in realising a more effective government. It identified three data terrains that together comprise the sources of information on government performance: (i) evaluations, (ii) programme PI and (iii) social, economic and demographic statistics (SEDS). It assigned to accounting officers the accountability for the frequency and quality of monitoring and evaluation information; the integrity of the systems responsible for the production and utilisation of the information; and it requires prompt managerial action in relation to M&E findings.
The GWM&E identifies the National Treasury as the lead institution responsible for programme PI. This is in line with its Constitutional authority for performance information and responsibility for prescribing measures to ensure transparency and expenditure control in each sphere of government as outlined in sections 215 and 216.
In 2007 the National Treasury issued the Framework for Managing Programme Performance Information (FMPPI). The aims of the FMPPI are to:
•define roles and responsibilities for PI,
•promote accountability to Parliament, provincial legislatures and municipal
Councils and the public through timely, accessible and accurate publication of
performance information,
•clarify standards for PI, supporting regular audits of non-financial information
where appropriate,
•improve the structures, systems and processes required to manage PI.
The Regulations on Programme Performance Information as part of chapter 5 of the Treasury Regulations are issued to ensure that financial and non-financial performance information underpins planning, budgeting, implementation management and accountability reporting to promote transparency and expenditure control towards economy, efficiency, effectiveness and equity in the use of public resources.
The Directorate: Performance, Monitoring and Evaluation developed an evidence based monitoring and evaluation information repository system (FSDOH-VIBE) that will support the development of monitoring and evaluation systems by various Institutions/Districts/Directorates/Chief Directorates.
It is a statutory requirement that, the Accounting Officer of a Department, Municipality, State Institution and the Chief Executive Officer of a Public Entity, must establish a Monitoring and Evaluation (M&E) System for his/her Organisation or Institution. Primary users of the M&E System will use these source systems to refine their planning and implementation processes (GWMES).
The Framework/Strategy describe the approach the Department will follow to create and operate an effective PM&E System that will produce credible, accurate evidence based performance information on an ongoing basis that are used to improve service delivery and sound governance.
Legislative and policy background is outline in Table 1.
TABLE 1: M&E LEGISLATIVE AND POLICY BACKGROUND
LEGISLATION AND POLICY
/RELEVANCE
The Constitution of the RSA (Act 108 of 1996, as amended)
/ Section 133 (3) (b): Members of the Executive Council of a province must provide the legislature with full and regular reports concerning matters under their control.The Constitution Section 215 and Section 216 Budget & Expenditure Management ensure information on inputs, outputs and outcomes.
Public Finance Management Act (Act 1 of 1999 as amended by Act 29 of 1999) and its regulations / PFMA underpins planning, budgeting, implementation management and accountability reporting to promote transparency and expenditure control. In this regard the PFMA requires performance monitoring and reporting.
Treasury Regulations (2002) /
Procedures for quarterly reporting must be established for the institution to facilitate effective performance Monitoring and Evaluation and corrective action.
The Government-Wide (GWM&E) Systems /Measurement of politically designated outcomes for accountability.
Framework for Measuring Programme Performance Information FMPPI (2007) /This framework aims at:
- Clarifying definitions and standards for performance information in support of regular audits of such information where appropriate.
- Defining roles and responsibilities for managing performance information.
- Promoting accountability and transparency by providing Parliament, provincial legislatures, municipal councils and the public with timely, accessible and accurate performance information.
Negotiated Service Delivery Agreement (NSDA) / Health Outputs according to the Negotiated Service Delivery Agreement (NSDA). The government has identified four strategic outputs which the Health Sector must achieve. These are:
- Output 1: Increasing Life Expectancy
- Output 2: Decreasing Maternal and Child mortality
- Output 3: Combating HIV and AIDS and decreasing the burden of disease from Tuberculosis
- Output 4: Strengthening Health System Effectiveness
FSEGDP (Free State Economic Growth and Development Plan) / Articulates the vision and mission of the Provincial Administration for the political mandate 2009-14
1.3. Audit of performance information
The Auditor-General has a constitutional mandate and, as the Supreme Audit Institution (SAI) of South Africa, it exists to strengthen our country’s democracy by enabling oversight, accountability and governance in the public sector through auditing, thereby building public confidence. The audit of performance information has been phased-in over a period of years and constitutes an evaluation of the policies, systems, processes and procedures for the managing of and reporting on performance against predetermined objectives.
Reporting will be in relation to material shortcomings in these processes, systems and procedures and which may impact on the public interest. These findings will be included under the section of the audit report titled Report on other legal and regulatory requirements.
1.4Target Group
The target group for this framework is:
- Political principals, managers and all staff in the department who are responsible for service delivery as well as rendering support for these services;
- Other Government departments;
- Other actors or organizations that need to be involved in the evaluation process, such as potential evaluators (including academics and other service providers), partners and non-governmental organizations.
1.5Alignment with other government M&E systems
- The Government Wide Monitoring and Evaluation System
The National Cabinet under the directive of the Presidency approved an
implementation plan to develop a monitoring and evaluation system for use across
government, known as the Government-wide monitoring and evaluation system
(GWMES) in 2005.
- The Public Service Commission
Besides the legislative frameworks shown to govern the PM&E strategy above, there are also policies, norms and standards within and outside the public health sector, with which the DOH has to comply. The Public Service Commission (PSC) is the primary government agency, which keeps an eye on compliance to this, as an example the Khaedu program can be mentioned. The PSC also identifies and assesses in all government departments, changes in several areas. It also gives support to various M&E initiatives and promotes new culture of M&E at all levels of the government.
- Office of the Premier Free State Province
The Office of the Premier has developed a set of Provincial key indicators in consultation with interest groups, and undertakes to design and establish a Province-wide monitoring and evaluation system based on these.