DRAFTDRAFT for CommentDRAFT

Paper prepared for MTTH/PME WorkshopsKathmandu, Nepal, 13-27 January 2002

“Beyond the Box:”

An Innovative Habitat for Humanity Paradigm for

Participatory Planning, Monitoring and Evaluation:

Measuring and Increasing Program Impacts with Appreciative Inquiry

Malcolm J. Odell, Jr., Ph.D., Technical Advisor, P/PME, AsiaHabitat for Humanity International

“Measuring Transformation through Houses” (MTTH) Program DRAFT of 7 January 2002

Building Success and Increasing Impacts through P/PME

“If you look for success, you find--and create--more success.” This is principle that has played an important part in Habitat South Asia’s “Measuring Transformation through Houses” (MTTH) program in its participatory planning, monitoring, and evaluation (P/PME) program that seeks to focus on what’s working, and how to create more of that. Using a combination of appreciative and participatory approaches, Habitat Affiliates are not only actively measuring their progress, but also helping accelerate positive change in their own organizations and for the families who benefit from Habitat Houses. These have, for example, helped the Galle Affiliate in Sri Lanka to develop its own computer MS Excel based tracking system to track mortgage repayments, to develop and computerize their own monthly monitoring system for reporting overall program achievements to their local Board of Directors, and – most important – to accomplish in less than one year almost everything they had dreamed of during a strategic planning exercise for the next five years. In a country that has build almost 1500 houses since it started 7 years ago, The Galle Affiliate itself has already has laid the foundations for 500 new houses and plans are underway to finish another 1000 houses during the coming year.

Inspired by these types of successes, the MTTH program has gone on to develop through participatory processes a range of innovative tools for P/PME that are now included in Affiliate Monthly Monitoring Reports, new homeowner surveys, low-cost housing studies, participatory evaluations, strategic planning, sustainability initiatives, and international indicators for measuring broad community impacts of Habitat programs beyond housing. On this foundation, plans are now underway for a new 5 year program to further develop and test new approaches to P/PME that may signal emergence of a new paradigm for the field of monitoring and evaluation globally in which the very processes used are empowering and actively contribute to creating positive change rather than passively measuring and reporting change as has been the tradition.

Women in Sri Lanka working on APA ‘Design.’

Accomplishments of MTTH Program to Date

Overall, the MTTH program, now only 2 years old, has laid out concrete revised plans to achieve its targets for the 3 yr. grant period as well as going beyond grant objectives to develop tools, techniques, and outcomes not envisioned during project design. Program activities continue to be implemented to identify, demonstrate and increase the broad community impacts of HFHI and its partners. These include conducting innovative and effective participatory evaluations of Affiliate programs, and training a diversified group of P/PME practitioners at the local and national level including homeowners, Affiliate staff and board members, and representatives from the community at large or partner organizations. These activities contribute to improving the institutional capacity of Habitat and its partners, increasing the organizational effectiveness of national and Affiliate programs through increased utilization of P/PME findings, helping ensure the sustainability, visibility, and impact of P/PME functions, and contributing to the overall financial sustainability of Habitat programs in S. Asia. For a summary of MTTH objectives and achievements, see Annex.

Linking P/PME and Appreciative Inquiry

These accomplishments in Sri Lanka and Nepal are due in part to Habitat for Humanity in the Asia/Pacific Area undertaking to develop, test, and implement a whole new way of carrying out monitoring and evaluation based on a two-pronged strategy that is both

  • Highly participatory and
  • Positive, appreciatively focused

The participatory strategy, specifically the 7-Step Participatory Evaluation, drawing on work begun in 1998 by HFHI staff representative of the global program areas and design made operational by Asia Pacific prior to the implementation of the MTTH program, uses Participatory Learning and Actiontools and techniques widely tested and disseminated through such organizations as Pact and SEEP.[1] These proven techniques have been supplemented by the integration of Appreciative Inquiry, or AI as it is becoming known.[2] While the tools of participation and participatory research are relatively well known, the use of AI in evaluation and monitoring is relatively new.

AI is an organizational development approach that has found its way into dozens of Fortune 500 companies that Habitat for Humanity is adapting and integrating into its other approaches to organizational development. The AI approach flies in the face of traditional monitoring and evaluation which in the past too often focused on what’s wrong and how to fix it, unintentionally generating or increasing anxiety, fear, and often multiplying rather than reducing the problems identified. The Appreciative Inquiry approach is being integrated into Habitat’s already ‘bottom-up’ participatory approach to evaluation that is now being used by local Affiliates many other countries beyond Sri Lanka and Nepal. Together these are already yielding results that are attracting attention in other countries and other organizations.

Already in the process of introducing ‘state-of-the-art’ participatory techniques into its P/PME operations, Habitat International began to take further steps to explore approaches that might contribute to the MTTH program’s ‘beyond houses’ Participatory Planning, Monitoring, and Evaluation (P/PME) initiative. A number of HFHI staff members, having become familiar with Appreciative Inquiry, became interested in grass-roots applications of the approach that had been developing and using for the past several years in other rural development programs in Nepal, one of the MTTH program’s three pilot countries. Recognizing the potential of using such tools within a P/PME framework, Habitat sought to bring that experience into the MTTH. During the Detailed Implementation Plan (DIP) meetings that initiated the MTTH program in South Asia, the Habitat team envisioned that if it engaged in a successful P/PME program with an appreciative twist, not only should it be possible to measure the positive impacts of its famous housing programs for families, communities, and the Habitat organization as a whole, but also be possible for its process of positive inquiry to directly help promote even more positive impacts. Measuring impacts is the bread and butter of PME, but using the PME process to directly increase impacts is breaking new ground.

To test this idea, the MTTH program initiated an AI training program for all new Habitat P/PME staff in both Sri Lanka and Nepal, drawing in particular on the Appreciative Planning and Action (APA) model that had been developed and tested with The Mountain Institute, the Kali Gandaki Hydropower project, and the Pact Women’s Empowerment Program.[3] This was done through a ‘learning by doing’ approach to the training of trainers that involved P/PME staff directly in running a series of APA Participatory Planning, Monitoring, and Evaluation Workshops with local Habitat Affiliates and national staff in, Nepal, during mid-2000. These sessions were used to identify the broad community impacts of Habitat programs from which quantitative indicators and measures were subsequently developed. A similar series of workshops was conducted shortly thereafter with several Affiliates and the national organization in Sri Lanka. Together these yielded a list of about 20 indicators from which seven were later selected for testing through field surveys in Sri Lanka and Nepal.

It was in such an APA workshop in Galle, Sri Lanka, which resulted in a 5-year dream that included a new office, homeowner education programs, religious harmony, and a wide range of income generation and fundraising activities. When the P/PME team returned just 7 months later, they met in the Affiliate’s new office, saw the fruits of their income generation and fundraising efforts, and heard of their new partnership with Sri Lanka’s Southern Development Authority for building 1500 houses over the next 2 years—a target only hinted at in their 5-year dream! [4]

By late 2001 the foundations of 500 of the houses planned by the Galle Affiliate had been laid. Families were also actively participating in an innovative savings program, “Save and Build,” that served as harbinger of new grass-roots strategy to reach the very poor. Together developments such as these alerted us that we had a process on hand that might be even more powerful than we imagined.

Toward an Appreciative “Bottom-up” Affiliate Monitoring System

The APA-P/PME impact assessment workshops conducted early in the program engaged homeowners, staff, local and national board members to share their discoveries of broad community and family impacts ‘beyond houses,’ dream of even greater impacts in the future, design concrete plans for beginning to realize those dreams, deliver concrete commitments and ‘action-now’ to start the process, and then develop their own indicators and tools for measuring impacts and reporting these to their own local staff and board members. As a result of these ‘bottom-up’ up initiatives, local affiliates in Nepal and Sri Lanka re-designed an existing program reporting format to become a self-monitoring format for them to measure their overall progress and reporting this to their own Boards and local stakeholders—like a self-administered monthly “health check-up.”

“Theology of the Hammer” combined with APA’s “Do it Now!” to promote volunteerism in Surkhet, Nepal. Local youth, JET STM team from N. America and Japan

The development of these Affiliate monitoring and evaluation systems began with the APA-P/PME workshops in several Affiliates in Nepal where a cross-section of homeowners, staff, and board members went through the APA process to come up with actual and anticipated impacts, indicators, and means for measuring them. One of these Affiliates, in the central hill town of Pokhara, then went on to design its own reporting format, with a comprehensive set of quantitative measures, drawing on the existing HFHI monthly narrative program report format then in use in Nepal. Besides the quantitative indicators, the format was unique in that it was designed by the Affiliate to report their own status to their ownBoard, rather than as a report to the National Office as had been the case previously, which would, however, receive a copy of the report. This format was shared with all other Affiliates in Nepal who, together with the National Office and P/PME team, provided suggestions for refining the format first developed by the Pokhara Affiliate. A similar process was conducted in Sri Lanka and the two countries shared outcomes to design a format then tested it with Affiliates in both countries during 2000-2001. This format was also circulated within HFHI’s regional, area, and international PME teams in New Delhi, Bangkok, and Americus for review and consideration for wider testing and incorporation into future participatory evaluations. Following extensive sharing and testing, a revised format, providing for further quantitative reporting of additional financial and program data, was introduced in Nepal in late 2001 and shared with Sri Lanka for proposed implementation in all Affiliates in both countries during 2002.[5]

Measuring Impacts beyond Houses:

The Development and Testing of International Indicators

While the APA impact assessment workshops resulted first in a self-assessment monthly monitoring system, there was more to be done. Many impacts identified could not be properly tracked on a monthly basis, nor would the results of a housing-driven program be identifiable through such frequent measurement. Two approaches were developed to address this situation, the first a short, one-page homeowner survey, and the second, a comprehensive International Impact Indicator Study. The initial homeowner survey was designed by the PME teams in both Nepal and Sri Lanka to measure a number of socio-economic variables identified by Affiliates but not suited to monthly monitoring. In Sri Lanka this took the form of a ‘Pre-housing Survey’ which was tested in several affiliates during 2001, particularly among those where the new ‘Save and Build’ housing program was being implemented. A similar survey was designed in Nepal to be administered by Affiliate staff when homeowners came in to pay their monthly loan installments. These were exchanged between the two countries and the resulting formats a very similar in terms of variables and measures. Implementation of this format has been incorporated in the Action Plans of each pilot country for the remaining of the grant period. A copy of the Sri Lanka ‘pre-housing survey’ is attached.[6]

The second approach involved the development and testing of a specific set of indicators of broad community impact through use of a questionnaire that measured changes perceived by homeowners in their lives and communities since building their Habitat home. From approximately 20-25 indicators identified by Affiliates in the APA workshops in Nepal and Sri Lanka, 7 were chosen which were shared by both countries and appeared to have both a high level of validity as well as a measure of universality based on Habitat’s global experience.[7] These indicators included:

Education of homeowner families, especially children

Health, sanitation, and nutrition of homeowner families

Economics and income generation

Community empowerment

Individual empowerment

Networking and partnerships

Peace and reconciliation

Sri Lanka and Nepal each developed and tested 3-4 of these indicators using 5-8 specific measures for each indicator in the form of questions which asked homeowners to compare their family and community circumstances before and after building their Habitat homes. Each country conducted a pilot survey with a sample of homeowners and others in several Affiliates. The two countries then exchanged their indicators, questions, and preliminary findings and then refined their questionnaires to take account of local factors. The revised questionnaires, including all 7 indicators and the supporting questions, was implemented on a cross-section of all households which had built their homes at least a year before in order to provide a perspective on changes that had or had not taken place since occupying their new homes.[8] Preliminary results from the pilot surveys suggest that positive changes have taken place around all seven indicators.

Initial findings from these surveys indicate that Habitat appears to be having a clearly positive impact at both the community and household levels. Homeowners and community members believe that Habitat works with poor and low-income families for building decent and affordable houses for poor those without proper shelter. In their view, Habitat has demonstrated and increased love and sharing of one another's burdens together. People believe that Habitat promotes overall community development. They perceive that the education of their children, health and sanitation, and incomes improve once they have built their new home. Many have improved their outlook on life and self-confidence, with some reporting that they have quit smoking or drinking and developed sound savings habits. Their environment and surroundings have become cleaner and use of toilets has increased. Among an extremely poor group of landless Chaudhari people in Nepal, for example, some never imagined that they could ever live in the type of house that Habitat has made possible, recognizing that this has brought about many positive changes affecting their entire lives. Awareness, volunteerism and action around community development, transparency, good management, peace and reconciliations has increased. Helping intentions among each other have been enhanced. They have become more mobile.

Supplementary Accomplishments of the MTTH Program

If the MTTH program is well on track toward achieving its objectives, there are also a number of accomplishments that were not planned for or anticipated in the original objectives and implementation plan. First among these is the degree to which the combined participatory and appreciative approach to P/PME has reduced the negative feelings associated with many traditional monitoring and evaluation systems. Affiliates are demonstrating higher than expected ownership of the P/PME process and results. Several Affiliates have developed their own P/PME systems, including spreadsheet tracking of loan repayments, and some are moving increasingly toward a standard of 30-day on-time repayments instead of the 90-day standard that was prevalent only a few years ago. Several Affiliates are showing progress in moving toward 100% on-time repayment. The APA process has proved important in many of these developments and, in particular, has been adapted for the development of a Sustainability for Local Fundraising workshop model that has been developed and tested in Nepal. This and related P/PME activities appear to be helping support a high degree of program innovation in Sri Lanka that includes ‘Building in Stages,’ ‘Save and Build,’ and special initiatives to reach the very poor and to eradicate poverty housing in entire communities. The process, through an AI adaptation called ‘positive deviance,’ is now being used in Nepal in an effort to identify those among the very poor who do have decent housing, to seek the best in housing among the poor, and to use this indigenous knowledge to help identify new and lower cost alternatives for house construction. A preliminary assessment by the Sri Lanka P/PME team has uncovered a positive relationship between housing for poor estate workers, their labor productivity, and the ‘bottom-line’ profitability of the concerned estates. Those estates in partnership with Habitat have proved to be among the most profitable in the country.