Harmonizing Management Sciences for Health Organisational Capacity Assessment Tools

harmonizing Management sciences for health organisational capacity assessment tools

Program for Organizational Growth, Resilience and Sustainability (PROGRES) Tool

with support from

the Innovation challenge fund (INCH FUND)

final project report by

HENRY KILONZO AND WYCLIFFE OUMA

table of contents

Acknowledgement……………………………………………………………………………

List of Acronyms

1.0 Introduction and Background

1.1 The INCH Fund Application

2.0 Methodology/Approaches

3.0 Project Achievements

3.1 Development of the Master OCAT

3.2 Development of the Organisational Nomenclature

3.3 Development of Indicators and Scoring Criterion

3.4 Administering the Tool

3.5 Data analysis and presentation

3.6 Reporting format

3.7 Summit Closure

4.0 Post Master OCAT Summit

4.1 Pre-testing of PROGRES Tool

5.0 Post Pre-testing the Master OCAT

6.0 Development of Master OCAT User Guide

7.0 ANNEXES

Annex 1: List of Summit Participants

Annex 2: Domains and Sub-domains

Annex 3: Issues Identified During the Pre-testing

Annex 4: The Final PROGRES – See separate attachment

Annex 5: The PROGRES USER GUIDE (See separate document)

Annex 6: The INCH Fund Proposal

Annex 7: Summit Agenda

Annex 8: Agenda of PROGRES Tool Pre-test

Annex 9: List of Participants for Pre-test

Acknowledgement

We wish to express our sincere gratitude to the leadership of Management Sciences for Health specifically Jonathan Quick and Larry Michael and the INCH fund committee for providing the resources to facilitate the development of the MSH Master OCAT that will facilitate organisational capacity assessment to enable provision of capacity building interventions for improved health outcomes.

Special thanks to the Summit members – Henry, Wycliffe, Sam, Marrieta, Andrew, Christine, Kahabi, Temesgen, Reshma, Jerusha, Judy, Daraus, Stephanie, Juliet, Atuma and Elizabeth for your sacrifice and commitment to ensure this innovative process is completed within the stated timeframes. We also appreciate the different project Directors from the five countries – Kenya, Nigeria, South Africa, Tanzania and Ethiopia for approving the participation of the Summit members.

Our colleagues from MSH global that we met in Pretoria – Monita, Kerry –Lee, we have not forgotten you. Your critical review of the tool provided the third eye for the missing details and overlaps in the tool. We say thank you.

The three organisations that played a key role in hosting us in their countries and also committed their time to be assessed as we tried the relevance of the tool - Africa Health Project (Abuja), Omega Foundation (Kenya), LENEPWHA and GROW (Lesotho), thank you very much.

These could not have happened without the moral support from our colleagues within the projects we work for – FANIKISHA Institutional Strengthening Project. We appreciate your role in keeping us motivated. Judy Seltzer, you were there for us in providing the motivation and technical support throughout. Thank you. ReshmaTrasi, thank you for creating time for the M&E team in Pretoria to review and provide inputs on the tool. This is great support that obvious came at a cost. We appreciate.

Finally, we acknowledge our efforts towards completing this assignment. The attainment of such a milestone needs endurance, sacrifice and high level of discipline. It was a high demanding task in addition to our core roles and responsibilities with high expectation in quality. Despite all the challenges we did not stop until the ship docked!

Henry and Wycliffe

List of Acronyms

CSOs / Civil Society Organisations
M&E / Monitoring and Evaluation
MSH / Management Science for Health
CEO / Chief Executive Officer
GAAP / Generally Accepted Accounting Procedures
OCAT / Organisational Capacity Assessment Tool
OCA / Organisational Capacity Assessment
PROGRES / Program for Organisational Growth Resilience and Sustainability
CLM / Centre for Leadership Management
BLC / Building Local Capacity
INCH / Innovation Challenge Fund

1.0 Introduction and Background

MSH has built an enviable reputation in health systems strengthening globally. Over the years, numerous tools and approaches have been developed to help guide the work. In this regard, MSH has become a pace-setter in leadership, governance and management in the health. Several projects in several countries where MSH operates in have developed different organisational capacity assessment tools. While they all assess the same areas, they are different in categories, wording and application methodologies. While these tools have been adapted several challenges prevail such as:

There are no common agreed upon approaches within MSH CLM for assessing organisational capacity development.
Different projects within CLM have varied organisational capacity assessment tools and approaches used to assess almost the same thing.
There is limited sharing by organisational development experts within CLM on the different OCATs.
Different projects use different OCATs to assess organisational capacity, and different metrics to gauge advances.
There has never been a joint CLM OCAT forum for all experts to brainstorm and identify areas of Harmonisation.
There lacks standard MSH online (comprehensive) OCAT, for use by MSH staff and also other interested organisations globally. FANIKISHA institutional strengthening project has developed one, but it does not include the contributions of other experts within MSH/CLM.

The above challenges have contributed to the following challenges within the MSH’s CLM:

Loss of time in adapting and sometimes creating new tools during new project start-up.
Different approaches and tools used to assess same thing, rather than perfecting a single suite of uniquely distinct tools.
Limited coherence in use of organisational capacity assessment terminologies and units of measurement.
Lack of common understanding on the “good number” of organisational capacity assessment domains or categories that needs to be in one OCAT.

Best practices require that ideas, or tools or practices, be tested and evidence provided to support their continued application. In 2013, there had not been a face to face dedicated meeting for the various OCAT authors to undertake an in-depth review of the existing OCATS with an eye toward creating a “Master” OCAT, and possibly a suite of materials for broad use across MSH. A virtual attempt had been made, but online discussions were also limited by time and technology. The experience of one of the FANIKISHA staff, Henry Kilonzo, when he interacted with colleagues in South Africa during the monitoring and evaluation workshop, provided an insight into how face to face discussions clarifies issues and help the consumers of knowledge to appreciate situations.

Based on this experience, Henry conceptualized the idea and invited a colleague Wycliffe to support him. Through the support of the INCH fund a forum would bring together all relevant but key OCAT experts within the CLM countries that have developed different OCAT instruments to assess the capacity of the target organisations. These countries include: Kenya, Southern Africa, Nigeria, Ethiopia, and Tanzania. The experts would develop a “master” OCAT, which would be accessible online by MSH/CLM and other organisations globally to contribute to identifying organisational needs, that, if addressed would contribute to improved health outcomes.

1.1 The INCH Fund Application

The team of two submitted an application to MSH through the INCH fund committee using the stipulated templates. Since this was a unique idea, a favorable response was received and the two colleagues, from FANIKISHA institutional strengthening project were awarded the grant - $ 10,000.00 commencing January – June 2014 and an additional $15,000.00 for pre-testing the tool. The award was aimed at using an appreciating enquiry to develop a Master OCA that when used would lead to:

Improved, harmonized “Master” Organisational Capacity Assessment Tool that can be broadly used across MSH.
Improved organisational capacity assessment results leading to clear capacity needs to inform technical assistance.
Reduced organisational capacity projects start-up resources since no more “development

2.0 Methodology/Approaches

The implementation of the project was based on “appreciative enquiry” where we acknowledged the important contributions of each project in creating an organizational assessment tool that helped MSH move towards its commitment of being a thought leader in institutional development. For the Master OCAT, several approaches were used. These included:

Identification of OCAT experts from the five countries – see annex 1 for details of participants
Collation of the different tools;
Development of review guide for the tools to establish commonalities and uniqueness. A template was developed to guide the analysis of all the tools by various country teams to highlight the following:

Clarity of domains
Mutual exclusivity of the scores
Units of measure and how realistic they were
Sufficiency of scoring criteria
Effectiveness of the application methodology
Clarity and mutual exclusivity of the final scores
Presentation of the results/scores
Successes and limitations of tool application
Resources and support required to administer the tool
Probability of impact on health outcomes and
Supporting tools like reporting templates

Desk review of all the OCATs from the 5 countries that included:

Kenya - FANIKISHA Organisational Capacity Assessment Tool
Southern Africa - Building Local Capacity Organisational Capacity Tool
Tanzania - Institutional Capacity Building Organisational Capacity Assessment Tool
Nigeria - Health Organisations Capacity Assessment Tool
Ethiopia - Leadership Management and Governance Organisational Capacity Assessment Tool

A 5-day forum to review the current OCAT instruments in terms of:

Language and definitions
Management, leadership and governance domains
Scoring criteria, including weighting
Use of data for decision making
Elements of a single “Master” OCAT
Automation of the “Master” OCAT
Packaging, branding and launch of the “Master” OCAT
Development of a draft Master OCAT

Pre-testing the draft Master OCAT
Finalization of the Master OCAT
Development of a use guide for the Master OCAT
Development of a marketing and communication strategy for the Master OCAT

3.0 Project Achievements

3.1 Development of the Master OCAT

A five day Master OCAT Summit was held in February 2- 7, 2014 in Naivasha Kenya. 15 experts participants representing five countries and MSH home office.

The objective of the summit was to:

•To review and understand the strengths, limitations and conditions for successful application of the current Organisational Assessment Tools being used by field projects in the five countries.

•To identify the best practices as well as redundancies in tools and merge elements where appropriate

•To create a Master Organisational Capacity Assessment Tool that will be rolled out to all of MSH for adaptation and application across projects

•To develop a marketing and communications plan to support the roll-out of the Master OCAT

3.2 Development of the Organisational Nomenclature

The Summit was opened by Dr. Daraus Bukenya, the MSH Country Representative, Kenya and the Chief of Party, FANIKISHA Institutional Strengthening Project. In his remarks, he emphasized the need to develop simple tools that are comprehensive enough to gather organisational capacity baseline data to facilitate capacity building plans. The tools should be linked to minimum benchmarks and should be used periodically to assess the progress made towards building strong organisations.

During the Summit, each expert made their presentations based on their tool highlighting about the tool, methodology, scoring, and limitations of the tool. The team asked questions on the presentations to enhance their understanding that would enhance harmonisation process.

During the Summit, participants identified organization and nomenclature that involvedidentification ofcore areas that were considered very crucial for any organization. These areas were named domains and their sub-domains. The following core domains were agreed upon:

The Core Domains – applicable to all organisations

Governance and Leadership
Resource Mobilization
Financial Management and Operations
Human Resource Management
Grants and sub-grants Management
Project/Program Management
M&E, Communications, and Knowledge Management

The second tier domains to be used based on the organisations mandate

Coordination and Collaboration
Advocacy, Networking, and Alliance Building
Service Delivery and Quality Assurance

Under each domain, the team identified the following sub-domains. See Annex 2 for the sub-domains under each domain.

3.3 Development of Indicators and Scoring Criterion

Participants agreed to adopt the 1-5 scoring criterion because of its mutual exclusivity and wider scope to give a picture of where the organization is at.The five levels were based on the organisational grown model. The meaning of the stages was as follows:

Fig 1: Scoring the Tool

Stage / Meaning of score
1 / No evidence of capacity
2 / Evidence of incompleteness
3 / Evidence of existence but not applied
4 / Evidence of existence but not fully utilized

Participants also identified the ideal practice/minimum standard of the sub-domain, key question to lead to various stage scores and verifiable indicators/means of verification to look for when assessing an organization. The figure below highlights an example of how the scoring should be done along a continuum of stage 1 – 5.

Figure 2: Sample Complete Domain

Domain: Finance compliance and Operations
Sub-domain / Ideal Practice / Key Question / Stages / Means of verification / Scores
Stage 1: Not present / Stage 2: In development / Stage 3:Present / Stage 4: Present, some application / Stage 5: Present, complete application
Financial Policies and Procedures / The organisation has written and approved Operational Financial policy and procedures known and refered to by all staff. / Does the organisation have operational financial policies and procedures known and adhered to by all staff? / The organisation does not have any financial policies and procedures. / The organisation has financial policies and procedures that are not documented OR financial policies and procedures need update. / The organisation has written financial policies and procedures, but the practice is not in accordance with the policies and procedures. / The organisation has written financial policies and procedures in the form of a manual, which is readily available to staff. The policies and procedures are in conformity with GAAP/IFRS, but are not consistently used. / organisation has established and operationalised policies, procedures, and systems exist guiding all aspects of financial management, readily available , and consistently practiced by all staff members and meet generally accepted accounting practices (GAAP). / Finance and operation policy / manual

3.4 Administering the Tool

The team agreed on the following methodology to administer the tool:

Facilitation
While the tool can be self-administered, the team recommends that the process is led by Facilitators
Minimum of 3 Facilitators, one of which must be from the organisations being assessed are required
2 days facilitators orientation workshop on facilitation skills is required, OCA tool (content and documents review)
The facilitator should use a facilitator’s guide as reference materials – see attachment 2
Pre-assessment:
It should be a participatory process involving one day alignment meeting with senior management team for:
Buy in and ownership
Identify change champions
Present OCA tool and requirements
Agree on the participants for the OCA based on the skills/levels/departments in the organisation. It should represent organisation diverse skills
Pre-assessment: Orientation of internal and external facilitators
Decide on the most convenient timing for the assessment taking cognisant of organisation’s calendar
As much as possible the venue should be at the organisation's offices
Internal facilitator or change champion to conduct orientation with staff and prepare evidence
Internal and external facilitators meet to conduct assessment
Assessment and consensus building
Assessment period needed is a minimum of 2 days and a maximum 3 days
Assessment should be evidence based to inform the consensus score – systems review first, then consensus building
It should be supported my MS -Excel or relevant platform to generate automatic and real time reports
It should be administered annually to review and measure progress

3.5 Data analysis and presentation

The summit member agreed on a data analysis and presentation of the analysis of the data as follows:

The subdomains under each of the sub-category can be weighed but should be flexible based on the assessment context. Scoring can involve decimals (one decimal place) or whole numbers. The tool would be automated (and also online) with the capability of generating dashboards.

3.6 Reporting format

Participants developed and agreed on a report template to help the assessors to document the findingsf the assessment. The reporting template is as below:

Cover page
Table of contents
Acknowledgements
Executive Summary
Introduction/background of assessment
Description of organisation being assessed
Assessment process
Definitions
Description of Methodology
Findings
Discussion of findings
Strengths in performance
Gaps in performance
Capacity building needs
Appendices
List of participants
Assessment agenda
PROGRES Tool
Capacity building action plan

3.7 Summit Closure

The Summit ended with the development of a Master OCAT for MSH that would be subjected to pre-testing to refine it further and make it more relevant to different organisations and the government. Dr. Daraus Bukenya, with support from Judith Seitzer, the

. “The commitment here demonstrated the true MSH values of vitality and excellence. This is a testimony to the MSH strategic roadmap and being a learning organization – Wycliffe Ouma