Fifth European Conference on Evaluation of the Structural Funds
Budapest 26/27 June 2003
Author: Robert Arnkil, Director of Research with co-author Timo Spanga, Senior Researcher, Social Development Company Ltd), Hämeenlinna, Finland
Evaluation, Dialogue and Learning in Multi-stakeholder Settings
The paper is presented in the Second Workshop Session, Evaluation and Capacity Building, Workshop 7: Learning and Using Evaluations
Abstract:
Problem Statement : The new designs of ESF programmes, exemplified by particularly Community Initiative Equal, have evolved into a more complex and challenging mode, combining developmental partnership building with thematic and transnational networking and learning. While presenting all the traditional challenges of evaluation – impact, comparisons, deadweight, etc. these new modes of operation present new methodological challenges for evaluation: What constitutes a good developmental partnership in a particular context? How to capture the evolution of developmental partnerships? Are they good learning networks, and, what in fact constitutes a good learning network? Are they effective in reaching institutional impact (mainstreaming) and not only transitional project/customer impact? How to capture the multi-stakeholder, or, indeed, the heavily stressed social innovation and empowerment aspects of these complex designs? How to facilitate learning across different contexts? On top of all this, evolution in such programmes happens quickly, and to a large extent tacitly. Is the traditional linear evaluation design – ex ante –ex nunc – ex post adequate to capture this “time compression” and demand for more real time learning and decision making?
Looking for Answers: There are no simple answers to these complex challenges. Drawing on several years of being the main responsible evaluator of ESF programmes in Finland, and also conducting transnational evaluation, the authors want to explore the possibilities of addressing the challenges described in the problem statement. The paper concentrates particularly on the challenge of addressing “real time” learning, building horizontal and vertical learning networks and dissemination in evaluation. The paper presents a set of methods called Emergent Dialogic Evaluation, an integrated approach drawing on constructivist, realist and knowledge management approaches in evaluation. The case examples explore the methods in the context of Community Initiative Equal and Objective 3 in Finland, and others.
Position and background of the authors
The presenting author, Robert Arnkil, director of research in Social development Co. Ltd. is responsible for the ongoing evaluation of CI Equal and Objective 3 (also in the former programme period). The author is responsible also for several other programme evaluations in Finland, for instance the Finnish National Workplace Development Programme ( Finnish acronym TYKE), the National Age Programme for Workers and also transnational evaluations, for example the evaluation of the European Agency for Safety and Health at Work (OSHA). Timo Spangar is a senior researcher in all of the aforementioned.
Correspondence: Presenting author: Arnkil, Robert, Social Development Company Ltd, address: Wetterhoffinkatu 4, FIN-13100 Hämeenlinna, Finland
e-mail:, phone:+358-50-5632843, fax:+358-3-6121262
EMERGENT EVALUATION AND LEARNING IN MULTI-STAKEHOLDER SETTINGS
Introduction:
This paper is divided in to two main parts. The first part, “Real –time learning settings”, wants to give the reader a practical feel of the methods and contexts. The second part, “What’s behind this” is a reflection on the background for the need for real-time evaluation and the contribution and limits of the methods presented.
We will start by briefly describing five different practical settings where we have used dialogue methods to promote “real-time” learning and at the same time gather information for evaluation purposes.
The first case is a dialogue workshop arranged in Helsinki in January 2003 for Developmental Partnerships (DP) of the Community Initiative Equal (CIE) and the Programme managers and coordinators in Finland. The case illustrates how a special dialogue arrangement we call “360-degree dialogue” can promote horizontal and vertical networking of projects and at the same time provide useful information for evaluators and Programme managers, and other stakeholders. The second case briefly explores a similar experience in the context of Objective 3 in Finland.
The third case is a dialogue conference that took place in Tampere, Finland in November 2002 for a broad set of stakeholders around building a sub-regional employment strategy and promoting the implementation of one on the DP:s in Finland. In this case a somewhat different method, called Future Dialogues, was used. The case illustrates how a special dialogue arrangement was used to promote commitment, creativity and networking of a broad set of key stakeholders in a sub-regional setting. The fourth case briefly explores a similar experience in the context of developing local employment strategies in another region.
Finally, the fifth case is a learning workshop process (“TuTori-project”) arranged for the Management Team of the Ministry of Labour in Finland in 2002-2003. The case illustrates, how, by special arrangements, the initial findings in evaluations could be used for learning in top management teams, and to work towards a new reflective knowledge management based work culture in the management of the ministry.
PART I : Real –time Learning Settings – Practical Examples
Case 1: “360-degree” Evaluation Dialogue with Developmental Partnerships of Community Initiative Equal in Finland
The first case is a dialogue workshop arranged in Helsinki in January 2003 for Developmental Partnerships (DP) of the Community Initiative Equal (CIE) and the Programme managers and coordinators in Finland. There were altogether almost 100 participants representing over 20 DP:s, Equal coordinators, Programme managers, representatives from different Ministries from Finland and other independent evaluators besides Social Development Co., the CIE Programme evaluator.
The case illustrates how a special dialogue arrangement we call “360-degree dialogue” can promote horizontal and vertical networking of projects and at the same time provide useful information for evaluators and Programme managers, and other stakeholders.
Let us first present a 360-degree model in a simplified form (figure 1). The model identifies six key dimensions, or themes for a dialogue: 1) Customer/citizen perspective 2) Internal functioning, or capability of the project 3) horizontal networking (both in terms of cooperation in actual service delivery/intervention and in terms of learning relationships) 4) vertical networking (how the different relevant levels of management and governance related to the project function (i.e. not only the immediate steering bodies of the project, but the broader context of management, like management and governance of a regional or sub-regional employment strategy) and 5) the time dimension, i.e. the developmental path of the project and the context.
Figure 1: The 360-degree model
The dialogue workshop was arranged as an open collective interview, where the evaluator, acting as the facilitiator for the dialogue, would invite representatives (5-7 maximum) from one of the DP:s to the front of the workshop to be interviewed as a group, while the rest of the workshop listen. The arrangement is described in figure 2.
The evaluator would ask questions about the 5 basic dimensions of the 360-degree model. Usually one would start with the customer/citizen perspective by asking the group for instance: “How would you describe your contact to your customers? How successful have you been in establishing rapport with your customers? How have your customers participated in your project? In planning? Launching? Redesigning? Etc. The point is to explore and elaborate a while on the customer perspective and also ask for evidence and indications.
Figure 2: Dialogue arrangement
Finally, after reaching a point of “saturation” (no new aspects seem to emerge, at least for the time being), the group would be asked to give a self-evaluation of the customer dimension (resembling the method used for instance by Fetterman in “Empowerement Evaluation”). Then the evaluator would turn to the audience, first the representatives from other DP:s, and ask them to reflect how their experiences perhaps are similar or dissimilar to that of their colleagues.
The dialogue would then proceed to the other dimensions – internal cooperation and capability, horizontal relations, management, developmental path…and each dimension would be also self-evaluated.
While one evaluator acted in the role of interviewer, another was in the role of a reflective partner and a note-taker. We also experimented with modern technique, so that the notes are reflected via computer and projector in real time on a big screen, where everybody in the forum could read it. The notes are made according to speakers and themes, so it gives a quick overview to how those present perceive the 360 degree context. The evaluation partner acts also in a reflective role, where her/his task is to keep an eye on how the dialogue is unfolding and helping to keep the other evaluator in a purely facilitative, interrogative role. The notes of the dialogue would be slightly edited (by noting the key themes) right after the workshop and then sent to all participants (and often to other stakeholders who might be interested in the matter).
The screen (and notes) would be arranged in the following way:
Figure 3: Real time big screen arrangement
Theme in 360 degree / Themes in dialogueCustomer perspective
A says: “I think we have succeeded in empowering our customers, because we used focus groups in designing our projects
B says: “I think there is still room for improvement to engage the SME’s” / Customers involved upstream
SME engagement
Internal functioning
D says: “xxx”
Etc.
Case 2: 360 degree method in an Objective 3 Context
A similar arrangement was used for a dialogue workshop for Objective 3 projects in March 2003 in Helsinki. There were about 80 people present representing 30 different Objective 3 projects in the region. Four case examples of the projects were used to stimulate dialogue, and a “tally” of self-evaluations about the success in the different dimensions was taken from the whole group at the end of the day. The relevance of dialogue in the Objective 3 context is probably even higher than in Equal, since Equal has a stronger design to invite cooperation in the first place, whereas in Objective 3 there is a tendency for projects to operate and drift apart.
The rationale of the 360-degree method
In every complex project or evaluation there is a multitude of actors and stakeholders. Each one has a potential “voice” in the project affairs. The project entity has vertical (steering) and horizontal (partnership and competition) dimensions, and one key challenge for any project is to position itself in this “360 degree” context. In fact, the key challenge for a project is to position itself so that it can realise and stabilise a new kind of future in a 360 degree context. Take the development partnerships (DP’s) of the present EQUAL Community Initiative programme, for example. The DP in itself is a complex network pattern, and is has a challenge of positioning itself in a particular context (where there typically is no shortage of others trying to position themselves). At the end of the day, the challenge of any project, including the DP’s, is to be able to transform the “temporary social capital” or learning network, existant in the DP-phase, into something more sustainable and institutional.
The rationale from an evaluation point of view is that the 360-degree dialogue provides interesting and useful data about the networking and learning aspects of the projects/ programmes, otherwise difficult to assess.
Case 3: Future Dialogues in Promoting Local/ subregional Employment Strategies
Let us now turn to another case illustrating a somewhat different arrangement. The second case was a dialogue conference that took place in Tampere, Finland in November 2002 for a broad set of stakeholders around building a sub-regional employment strategy and promoting the implementation of one on the DP:s in Finland.
There were altogether over 100 participants representing NGO’s and third sector (associations of unemployed, disabled groups, etc), labour unions, employees organisations, business companies, municipalities (managers, politicians and shop floor), state services (public employment offices), the Equal DP of the sub-region (managers and project coordinators) and other project related to domestic and European Union, social scientists, educators, etc.
In this case a somewhat different method, called Future Dialogues, was used. The case illustrates how a special dialogue arrangement was used to promote commitment, creativity and networking of a broad set of key stakeholders in a sub-regional setting. We would call the different stakeholders present “voices”, and would arrange the setting in a similar way than in the first case, namely, inviting a “voice” (a group of people, 5-7 maximum) in turn to be interviewed. Only this time the questions would be different.
Figure x: An example of a set of key “voices”
- Customers, beneficiaries
- Partners (Private companies, 3rd sector, public services)
- Benchmarking companions (other DP’s, other relevant learning partners)
- Regional/ local decision makers (managers and politicians)
- Programme managers and coordinators
- Specialists, consultants, educators, scientists
The evaluator (team) would act as a facilitator, or “stage manager” for the forum in a special way. In conducting the Good Future Dialogue, the evaluator would ask the representatives in the forum, the “voices”, to imagine that we have (“miraculously”) moved to the future, say 1-2 years ahead (depending on an appropriate time-span), and that the endeavour or project at hand has been successful. A good platform for future development has been reached. Now the only task of each “voice” is to remember what has happened, to remember the future.
The evaluator would say something like: “I have heard, and read in the newspapers, that really good progress has been made in this project… could you tell me, each in turn, what makes you particularly happy about the project…”
Three questions would be asked, in turn from each member of the group:
1)What makes you particularly happy about the progress in the project from your point of view? What was your contribution to this positive development?
2)Where did you get support/ help/ important assistance to achieve these results?
3)Where you worried about something a year ago – and what helped to lessen your worries?
The evaluator would pursue the question with a few additional questions, depending on the answers, with the intent of getting as concrete a picture as possible, what has happened and who did what. Careful attention would be given to the fact that we are in the future and that progress has been made. So it’s not “ifs”, it has really happened. The evaluator would also take care that he/she uses the words of the interviewee as far as possible in summarising what has been said.
After interviewing the “voice” briefly (about 30 minutes), the evaluator would open a brief round of reactions from the audience, asking does someone remember things differently, or do they want to add or clarify something. Then another voice would be invited. They would be asked to remember the future from their point of view, with using what they already heard as a source of information and inspiration. So cumulating and saturation of themes is invited.
A limited number of voices – usually a maximum of 4 – 6 for practical reasons – would be heard, which takes about 3-4 hours, depending on the number of people in the groups, and also the swiftness of saturation in the discussion.
We can see that the three main questions map out some key aspects of the project and the actors:
1)Good future: The perception of the “voice”, from her/his viewpoint, what might be a good result in a year (or some other appropriate time-span). This gives the possibility of others to get to know each and every one’s perspectives and aims.
2)Support: This maps out the key resource network, as perceived by the “voice”, and gives the possibility of all present to see how each and every one (and others) is perceived as a resource (or not).
3)Worries: This maps out the present day worries and anxieties and obstacles – but does not end there: the voice is asked to identify possible solutions to avoid pitfalls.
Adding a time-dimension (ex ante-ex nunc and ex post in “real time”)
Depending on the circumstances, a minimun of two other fora/workshops would be arranged for the same groups, to provide a time-dimension for self evaluation and outside evaluation. This also the case for the Tampere group, where dialogue conferences will be arranged in 2003 and 2004. At a mid-term point (preferably as close to the time-span of the future perceived in the first workshop) the “voices” would be invited to assess (via self-evaluation) progress against the Future they described earlier. In this workshop various methods to assess the degree of progress would be used (i.e. rating scales, etc.). Then a new prediction about the possible future would be made with a Future Dialogue, now for the remaining period of the project. At the end of the project, a similar process would be undergone, with making again a journey to the good future, now extending to the mainstreaming period. The question would then be about successful mainstreaming and how it took place.