project: Impact Evaluation

Clent: Impact Evaluation

Summary

  1. Introduction

The purpose of this document is to present findings, key conclusions, lessons learnt and recommendations of an Impact Evaluation of two similar projects that have been implemented by the Opportunity Bank of Malawi (OBM) over the past three years. The two projects are the Scottish Government ‘Sustainable Livelihoods in Nsanje’ Project and the Department for International Development (DFID-UK) ‘Sustainable Livelihoods in Southern Malawi’ Project. The Impact Evaluation (IE)[1] took place in December, 2012.

The IE team was led by Bright Sibale from the Centre for Development Management (CDM). The team worked closely with Opportunity International (UK) and Opportunity Bank of Malawi[2], especially in developing methods and tools for data collection. In terms of purpose and use, the IE was final project evaluation for the Scottish Government Project and a midterm evaluation for the DFID Project.

The Scottish Government funded project started in 2010 and will be completed in March 2013. The DFID supported project scope is similar to and scales-up the Scottish Government Project. The DFID project began in January 2012 and is expected to be completed in December 2013.

In addition to Nsanje, the project is implemented in Mangochi, Blantyre and Mulanje districts, which are all in the Southern Region of Malawi.

  1. Objectives of the IE

The purpose of the IE was to evaluate how far both the projects’ intended outputs and results have been achieved in relation to targets set in the Scottish Government project plan and the DFID project logical framework. Specifically, the IE was to assess the five evaluation criteria of relevance, efficiency, effectiveness[3], impacts efficiency, and sustainability of the project. Key evaluation questions were as follows:

Relevance: were the programme’s objectives consistent with the needs of poor people and the policy of the Government of Malawi?

Effectiveness: the extent to which objectives have been achieved. In the terms of reference, effectiveness was tasked was, "to evaluate progress of the project -compare performance with pre-set targets and judge whether the project achieved its main goals in the time frame

Impact: what was the totality of impacts (negative or positive) of the programme on improving the livelihoods of the poor? And provide recommendations for the project, extracting lessons learned.

Efficiency: were the costs of the intervention justified by its results and were resources used efficiently?

Sustainability: what is the longevity of results achieved by the programme after funding is terminated?

  1. Methodology

In terms of methodology, the IE used both quantitative and qualitative methods to assess relevance, effectiveness, efficiency, impacts and sustainability of the project. Data collection was done using key informant interviews, focus group discussions, literature review and a client survey. The client survey interviewed 440 OBM clients from seven outlets, namely: Mangochi (Mangochi District), Kabula and Ndirande (Blantyre District), Mulanje and Limbuli (Mulanje District) and Nsanje and Fatima (Nsanje District). The table below shows distribution of respondents by outlet and by type of data collection method.

Outlet / Client survey / Focus Group Discussion / Key Informant Interviews
Ndirande and Kabula / 61 / 2 / 5
Nsanje and Fatima / 254 / 4 / 8
Mulanje / 61 / 4 / 3
Mangochi / 64 / 3 / 4
Total / 440 / 13 / 20
  1. Summary findings
  1. Macroeconomic Context

The projects have been implemented in a very unfavourable macro-economic environment, particularly between 2012 and 2013. In 2011-2012, Malawi was rocked in governance problems at national level. Most donors withheld their budget support, which resulted in an economic meltdown characterised by forex shortages, fuel scarcity, rapid increases in prices of goods and services. These macro-economic shocks negatively affected OBM clients businesses, hence affected achievement of project objectives. For examples, OBM outlets visited that operate mobile vans (Mangochi, Mulanje and Nsanje) had problems to buy fuel and therefore could not consistently service their mobilise van clients; most loan clients also complained about scarcity of raw materials and increasing prices, while those who operate cross-border businesses had problems to buy forex, which negatively affected their businesses.

  1. Relevance

The IE has found that the two projects are relevant to achieving Millennium Development Goals (MDGs) at global level, to the Malawi Growth and Development Strategy (MGDS) and sector level policies at national level, to local District Development Plans as well as to needs of the community. For example, the MGDS has identified "improved access to financial services" as a priority issue under the economic governance sub-theme within the broader theme of improved governance (GoM, 2012). Particularly, the two projects are critical in attaining economic empowerment of local communities, especially women, by providing financial services to the rural poor. The IE found that the two projects are relevant to the needs of the people and the country, particularly those in the Southern Region of Malawi where poverty levels are quite high. For example, according to the FINSCOPE Study of 2008, over half (55%) of the Malawi adult population in Malawi are financially excluded i.e. they manage their lives without using any kind of financial product (formal or informal). The results from IHS3 indicate that in Malawi about 14 percent of the households had some interaction with the credit market, 8 percent of whom successfully obtained at least a loan, 5 percent of the households tried to get a loan in the last 12 months but were turned down and 2 percent are still waiting for a response on their loan applications, (NSO, 2011). The IHS3 further shows that the proportion of loan recipients in the first four consumption quintiles increases from the lowest per capita consumption quintile (5 percent) to 11 percent in the fourth per capita quintile, which suggests that better-off households are more likely to have obtained a loan compared to worse-off households. This also means that by targeting rural poor, therefore, OBM projects promote financial inclusion for poor people. At regional level, the highest proportion of persons who accessed loans is observed in the central region at 11 percent, followed by the southern and northern regions at about 6 percent. This again shows that implementing projects in the Southern Region is contributing to and addressing access to loans for poor people.

  1. Effectiveness

The projects have met the majority of their numerical targets and not met other targets. In general, however, the view of the IE is that although some target numbers have not been met, the overall picture is that the projects have been successful, particularly in economic empowerment of women and men. Through the two projects, OBM supported 7 outlets which are currently serving a total of 23,070 savings clients and 7,531 loan clients in all the 7 outlets that have been supported by the two projects. In terms of access to financial products and services, which related to Outcome 1 of the projects, the IE found that their access to financial products has increased in all outlets. The OBM products and services include savings accounts, loans, financial education, ATMs, mobile van and many more. Poor people living in the areas surrounding OBM branches now have options to transact with the bank thereby improving financial inclusion of poor people, including those living in highly inaccessible places such as Fatima in Nsanje, Thekerani in Thyolo and parts of Mangochi. Such remote areas have traditionally been "no go zones" for banks for a long time. The projects have also economically empowered clients’ households by providing loans, which have been used to expand clients businesses, thereby increasing households’ farm and non-farm incomes. In addition, the projects have helped to create employment through self employment and as well as employment of people working in businesses belonging to OBM clients. Furthermore, the projects have diversified the sources of loans for clients.

  1. Efficiency

The projects have been efficient in that they have achieved many impacts that will remain with the communities and particularly clients for quite a long time. The projects have established 7 outlets, where OBM now operates. The branches will continue to offer financial services to the poor long after the funded phase of the programme expires. Individual and focus group interviews with clients reported that OBM accounts are easy to open as they do not require too many procedures, the loans have a longer grace and repayment period, liquidity, transactions do not require too much writing. In almost all branches visited, clients also reported that OM staff approach them in friendly manner. In Fatima many clients stopped operating their accounts at a SACCO and opened them with OBM due to high interest rates, limited liquidity and unreliability of the services. Hence more and more customers will continue to open accounts with OBM, due to its flexible and participatory approaches to provision of financial services. Although, not fully developed, partnerships with other organisations, particularly with Government and Non-government organisations have improved the efficiency of project implementation. Through collaboration, OBM has attracted new clients, with limited cost because the partner organisation (or referring organisation) will already have trained the groups, so initial investment in this case has been served.

  1. Impacts

Impact was defined as the totality of negative or positive impacts on financial, livelihood, food and nutritional and education aspects amongst OBM clients. The major impacts observed were as follows:

Access to financial services

  • With OBM opening outlets in the seven areas, the use of informal loan products and services, such as Katapila[4] has reduced from 5.4% to less than 1%. Katapila is expensive and does not respect the dignity of a human being and often leaves borrowers in perpetual poverty[5]. Another important impact from the client survey and shown in the figure is that the use of formal banks has increased from 1.9% before OBM to 62% after OBM opened their outlets. This also suggests that most clients are first-time bank account holders, which means the project has increased financial inclusion by the poor.

Access to capital

  • Use of own income to finance businesses has also reduced from 40% before OBM to 23% after OBM opened the branches. In addition, business financing from wages and salaries (which are part of personal income) has reduced from 14% to 9%. Given that own income should essentially be used to support household livelihood day-to-day expenses, particularly for the low-income earners, who hardly earn enough to meet their requirements, this impact is important because it frees and allows the households to use the own income for day-to-day livelihood expenses, while business financing is taken over by the banks.
  • The percentage of clients who had no external sources of capital to finance their businesses[6] before OBM came into their areas has also reduced from 17% to 13%. This might mean that with arrival of OBM, more community members are learning from OBM clients to use their financial products and services.

Improved savings behaviours

  • In terms savings, the IE found that the projects have helped to promote a savings culture in the impact areas. For example, about two thirds of all clients interviewed (64%) had never had an account with any bank before they opened an account with OBM, while the remaining 36% of clients had an account before. Of those that had no account with a bank before, 60% were females and 40% were males. The mere fact that these opened an account with a formal bank means that there is a new behaviour to start transacting with formal banks.
  • Further, an important impact of the project is that clients are now using formal ways of savings other than informal and unsafe ways of saving money. The IE found that before OBM opened the outlets, half of all clients interviewed used to save money at home. This behaviour has been reduced significantly down to 2% at the time of the survey. Similarly savings using other banks has also reduced from 33% to 5%.

Reduced income variation and vulnerability

  • Another important impact is that clients who depended on paid job as the main source of income reduced from 16% before to 15% after OBM opened accounts in the outlets. Most paid employees were teachers and extension workers and key informant interviews with them showed that they are now more involved in businesses, hence have more disposable income. The majority of loan clients (92%) depend on business as their main source of income. Non-loan clients depend on two main sources of income, which are business 47% and paid job 41%.

Job creation

  • The projects have been effective in creating jobs for the rural people. Of all clients interviewed, 35% employed some people, while the majority of 65% did not employ any other people. About 40% of clients supported by the Scottish Government had employed some people compared to 33% of DFID clients, primarily because the former project started earlier than the later.

Improved access to agricultural inputs

  • In terms of food security, the projects have helped to economically empower households so that they are able to rely on food purchases from the market as the main way of sourcing their food. Traditionally most households in rural areas depend on own production as the main source of food. The IE shows that purchasing of the main staple from the market is more used (64%) compared with own production (46%).
  • The IE also found that clients (84% of those who received coupons), used their business income or savings to buy the subsidized fertilizer or seed during the 2012/2013 agricultural season. In addition, about 44% of all clients interviewed, reported that they used part of their business income to buy some commercial fertilizers, which are very expensive on the market.

Better food security as percieved by clients

  • About 59% of those who reported that they had seen some changes in their food security, indicated that they are now (after OBM), eating more nutritious foods and are eating two or more meals a day respectively. This collates well with the fact that clients reported increased business income, use of business income to buy food and improved access to business loans.

Impact on education

  • The average number of children of school going age rose by 3% from 2.32 to 2.38. In our view, based on our experience in the education sector in Malawi, this is a very high level indicator and often, it cannot be moved by one project or one institution. It requires longterm efforts to address it from a range of players.

More joint decision making between spouses, less unilateral decision making

  • Before they joined OBM, 63% of female and 65% of male loan clients made decisions by themselves, while joint decision making was reported at 25% and 23% by male and female loan clients respectively. The data shows that joint decision making has increased from about 24% to 35%.
  1. Sustainability

The IE found that the two projects have been implemented by OBM, which is a commercial bank and operates on market-based development strategies. The projects have been implemented to complement the expansion of OBM branches and have supported implementation of OBM business plan. The projects have not been implemented in a full projectised arrangement, which would have otherwise compromised sustainability. Staff members who implement the projects in the project outlets are members of staff and have same conditions as with other staff. The branches will continue to offer financial services to the poor long after the funded phase of the programme expires. More and more customers will continue to open accounts with OBM, due to its flexible and participatory approaches to provision of financial services. Although, not fully developed, partnerships with other organisations have improved the efficiency of project implementation.

  1. Challenges faced by the project

The following issues negatively affected the performance of the project and threaten the sustainability of the projects: the unfavourable macro-economic and political environment which led to Government to devalue the kwacha and increase base rate, which made loan products expensive and reduced the real value of the loan, the small loan sizes, the low uptake of IMC and SME loans due to collateral requirements, delays in processing loans, high cost of borrowing, inadequate staff incentives, project set-up and organization, weak operational support to outlets and weak high level indicator monitoring.

  1. Key recommendations

The IE recommends that OBM should review delivery mechanisms and marketing of its loan products, consider changing the project set-up and management, speed up reforms and strengthen human resource management policies to motivate staff members, strengthen partnerships and networking with Government and Non-Governmental Organizations, strengthen operational research and documentation and finally, strengthen HIV and gender mainstreaming and climate change issues. For each recommendation, the IE has identified specific actions that should be acted on to implement the recommendation.