External evaluation of the European Institute of Gender Equality (EIGE)

INCEPTION MEETING – 10-11 September, 2014

EIGE premises, Gedimino pr. 16, Vilnius

Participants (alphabetical order):

Steering Committee

Daniela Bankier, European Commission DG Justice Dir D- Equality

Helén Lundkvist, Ministry of Education and Research, Sweden

Pauline Moreau, Department of Justice and Equality, Ireland

Jo Morris, Independent policy adviser, United Kingdom (representing ETUC)

Anu Sajavaara, Service Sector Employers, Finland (representing BusinessEurope)

Käthlin Sander, Ministry of Social Affairs, Estonia

EIGE

Virginija Langbakk, Director, EIGE

Ruben Lombaert, Programming Officer, EIGE

Thérèse Murphy,Head of Operations, EIGE

Merle Paats, Administrative Agent: Statistician, Project Manager of external evaluation, EIGE

Contractors

Vassilis Alexandris, Deloitte Consulting, Belgium

Ivana Cenerić, PPMI, Lithuania

Rimantas Dumčius, PPMI, Lithuania

Anu Laas, Laas & Laas OÜ, Estonia

Laura Peciukonytė, PPMI, Lithuania

Donatas Pocius, PPMI, Lithuania

Hilde van de Velde, Deloitte Consulting, Belgium

Minutes

Introduction

Ms Langbakk greeted all participants and briefly introduced each member of the Steering Committee, welcoming the representation of different groups of EIGE’s stakeholders, partners and beneficiaries. The contractors briefly introduced themselves.

Context and objectives of the evaluation

The main objective of this evaluation, according to Ms Langbakk’s presentation, wasto assess the achievement of objectives and the impact of EIGE on the promotion of gender equality and, based on the evaluation results, consider whetherthe activities of the Institute could be extended, following the Articles 20 and 21 of EIGE Establishing Regulation. For this reason, the evaluation should producedetailed future-oriented recommendations on how its activity could be improved and how the evolving needs of its stakeholders and target groups could be better addressed. Having in mind that EIGE was in its establishment phase in the period of evaluation, an example of performance measurement indicators which EIGE could use in the future also needed to be made by the evaluation team. This needed to be taken into account when assessing the results and impact of EIGE, as many of its activities were still in their initial phase.

Ms Langbakk reminded that the evaluation in particular needed to take into account the changes in policy context since the previous evaluation efforts of EIGE (feasibility studies and ex ante evaluations). The evaluation also needed to carefully reflect the current actual needs of EIGE’s stakeholders and beneficiaries, focusing also on whether the activities of EIGE matched its objectives and foreseen tasks.

The evaluation also was seen by the Steering Committee as a good opportunity to clarify certain issues concerning the functioning of EIGE:

  • The evaluation should assess to what extent EIGE’s core mission, understood as a centre of competence and research on gender issues in the EU, is fulfilled. Does EIGE concentrate on its core activities? Which activities have had the biggest impact to promote the equality between women and men and which ones did not match the objectives, or had no clear impact or added value? As far as potential financial consequences of possible changes are concerned, the evaluation should not only assess if new or less activities would cost more or less, but look at how to improve budgetary decisions so as to improve efficiency and effectiveness of activities (strategic planning, performance-budget).
  • To what extent are Member States the clients of EIGE (as primary orientation of institute is European level institutions, but in fact EIGE carries out work for Member States and receives their requests)?
  • Is EIGE a centre of expertise / excellence, a research centre, an information platform, or an advocacy body? How well does it perform in networking and communication?
  • Should EIGE focus more of challenging the social norms to have a more equal society by identifying trends, problems and areas for action?
  • The evaluation should put emphasis on working methods to assess if things are done in the right way. It should consider if EIGE could benefit from changes of working methods, including fewer meetings, less administrative burden to the benefit of more quality discussion on substance (including at Management Board meetings). Taking the example of gender mainstreaming, it should be assessed if the network needs to be kept, in particular if it simply serves as a forum for the presentation of published reports.

In addition, it was underlined that EIGE as a European agencyshould serve primarily the EU level institutions, thus the evaluation should be carried out taking into account this perspective. It should assess to what extent the Commission’s priorities are taken into account in EIGE’s activities, for example to what extent specific advice on the Gender Equality Index was taken into account, to what extent advice from the Management Board is considered by EIGE, if and how comments on Work Programmes were taken into account. Besides, the current system of rotation in the Management Board should be assessed, taking into account the issue of “discontinuity” of the Management Board and the aspect of how to keep Member States who are not members involved and informed. Ms. Jo Morris also paid attention to the fact that social partners should be incorporated in the management board and that EIGE cannot be seen separately from the world of work. These aspects should be also discussed in the evaluation.

The evaluation should also have a critical look at indicators: do they help to assess impact, do they measure and assess performance properly? It should be seen if the EIGE's annual reports could benefit from indicators which would capture the essential substance of the strategic goals of EIGE instead of measuring simply outputs irrespective of their impact. EIGE clarified that a formal introduction of indicators will take place following an agreement within the network of agencies on this topic. This evaluation is also an opportunity to look at the future role of EIGE, in light of a future revision of its founding regulation.

Role of the Steering Committee

Steering Committee represents main stakeholders and beneficiaries of EIGE, and includes current as well as former members of the Management Board. Its main role is to guide the work of the contractors during the evaluation. The contractors are expected to have regular meetings with the Steering Committee, provide regular feedback (electronic monthly updates, especially on outstanding issues)on the progress of work, decisions made, risks encountered and planning of deliverables. Monthly updates could be used as an early warning system.The contractor is also expected to provide answer to any question or request within three working days. Ms Paats was named as the main contact person in EIGE facilitating the delivery of feedback between the contractors and the Steering Committee.

The policy context of EIGE

Ms Langbakk further presented the policy context of the Institute. She mentioned that the creation of EIGE was not based on a solid consensus. EIGE was created in the same period as a large number of other new EU agencies. A political decision was made to set up EIGE in a new Member State, making its relationship with other European institutions a particularly interesting case. The Institute was set up at the very beginning of the global economic and financial crisis, which also affected its establishment.

EIGE is a European agency,thus its primary task is to serve European policy making (the Commission, Council and Parliament). EIGE is anindependent agency created to provide policy making institutions data and other evidence for policy making. Thus in the evaluation it is important to analyse whether the Institute has succeeded in creating the conditions to meet European priorities. Additionally, it is important to explore whether synergy effects with other EU level institutionshave been created. It is particularly important for the evaluation to assess whether there is an overlap of EIGE’s work and the European Commission’s work, as well as the work of other EU decentralised agencies on gender equality. It is vital to find better ways of working together and to divide the priorities among different areas.

Early stages of EIGE’s functioning were complicated and its establishment was lagging behind schedule. EIGE also faced communication issues, as its experts were unable to sufficiently connect with the research going on in the broader field of gender equality (outside the European Union policy context).

At the moment, a few different European level trends are noticeable,which are of importance to EIGE. The process of consolidation of the decentralised agencies has been launched, and aproposal of merging EIGE with Fundamental Rights Agency was put forward in the Parliament, although this was not voted for. EIGE is also in the midst of changing European priorities on gender equality. Gender equality is regaining some focus in the European agenda, and the issues such as the economic engagement of women and availability of childcare are likely to become important in the coming years.

EIGE’s work at the national level is also challenging. The national ministries and beneficiaries do not always know what priorities they should take actions on and what methods they should employ.

EIGE intervention logic

Ivana Cenerić from PPMI made apresentation on the contractors’ understanding of EIGE’s intervention logic.This was followed by a detailed discussion.

The participants of the meeting agreed that during the evaluation it is important to look at theimpact of the Institute on the promotion of gender equality and identify how EIGE contributes to it and to what results it can be attributed to. However, it should not be forgotten that EIGE is only one of the many players in the wide and diverse field of gender equality. Thus it was difficult to attribute measurable long-term impacts (e.g. reduction of gender stereotypes) to the work of EIGE. This was particularly important having in mind that the evaluation primarily encompassed the set-up phase of the Institute. The evaluation team was therefore advised to attribute the right level of expectations on EIGE and to focus on how successful the Institute was in meeting the needs of the key stakeholders and beneficiaries, delivering its first outputs, products and services. Potential questions for analysis here included whether EIGE’s activities corresponded to its priorities, how they helped shape European gender equality policy, what were the main uses of EIGE’s work etc. Also, the question on the quality of EIGE’s outputs should be also addressed. In interviews stakeholders should not only be invited to make a broad assessment of outputs, but to demonstrate how they were used and if any aspects should be improved.The immediate impacts of EIGE, on the other hand, could be estimated more easily. Particular importance could be put on raising awareness about the issues of gender equality among the citizens of Europe as well as EIGE’s stakeholders, beneficiaries and partners, which could be measured e.g. through exploring whether EIGE studies are mentioned in policy documents or academic literature.

The Steering Committee agreed that the evaluation team’s presentation of EIGE’s activities was correct, but the identification of gaps of information on gender equality were one of EIGE’s work inputs rather than outputs.

Stakeholder structure of EIGE

Donatas Pocius from PPMI made a presentation about EIGE’s stakeholder’s structure, again followed by a discussion.

The reaction from Ms Bankier was that the evaluators needed to provide a clearer definition of what was meant by their definition of stakeholders. It was suggested that “key stakeholders” should be clearly defined and distinguished from “other stakeholders”. The research approach should be tailored to particular circumstances, roles played by stakeholders, their relations with EIGE. The evaluator should pay attention to potential conflict of interests as a considerable number of EIGE’s stakeholders are also supplying services to EIGE and / or benefiting from a range of activities or services by EIGE. Finally, a focus on “key stakeholders” should be also ensured during the evaluation. Ms Bankier argued that the key stakeholders of EIGE were the European Commission, European Parliament and Management Board, whereas the other groups interested in the field of gender equality and presented as EIGE’s stakeholders by the evaluation team should rather be called its beneficiaries, clients, or partners. It was agreed that the interpretation on which groups could be called stakeholders would be reviewed by the evaluators after it receives EIGE’s own description of its stakeholder structure.

The Steering Committee commented that the entities missing from the structure of EIGE’s stakeholders, beneficiaries and partners included COFACE, CEEP, as well as a wider list of European NGOs, social partners, family organisations, and national level organisations.

The final remark from the Steering Committee was that the evaluation should analyse the intensity of contact and cooperation between different organisations and EIGE. Based on the measured intensity of contact, the evaluators could focus on institutions which have more established contacts with EIGE and which are dealing with research related to EIGE issues. EIGE’s relationship with the Council of Europe, which was developed from the very beginning of EIGE’s activity, also needed to be explored.

Discussion of evaluation questions

Hilde van de Velde and Vassilis Alexandris from Deloitte Consulting as well as Rimantas Dumčius and Donatas Pocius from PPMI reviewed the evaluation matrix and evaluation questions according to each evaluation dimension. The Steering Committee had an opportunity to react to the presentation of each topic of the evaluation.

Efficiency

The comment from Steering Committee was that the evaluators should analyse the composition of EIGE staff (e.g. ratio of administrative vs. content staff,geographic coverage, coverage of thematic fields / specific required qualifications). The reasons for this structure could also be explored, e.g. how this is the result of regulations applicable to EIGE or the lower salaries paid to its staff (compared to the other agencies / European Commission). The effect of lower salaries on the ability to attract qualified experts and the quality of outputs produced.

The questions 6.1. and 6.5. were dealing with the stakeholders’ awareness about EIGE’s organisational structure, roles and responsibilities. The remark of the Steering Committee was that not all stakeholders needed to be aware of EIGE’s internal organisational matters and therefore only the awareness of stakeholders to whom this is relevant needed to be considered. In the meantime, the evaluators could rather analyse the clarity of division of roles and responsibilities among EIGE’s staff.

In addition to the questions addressed in the evaluation proposal, the Steering Committee defined the following issues to be explored:

  • Does the Management Board deal with the right issues?
  • What is the interaction scheme among different bodies of EIGE (Standing Committee, the Management Board, and the Experts’ Forum)?
  • Is it efficient to have a separate Standing Committee?

Finally, another remark from the Steering Committee was that in the presentation of EIGE’s organisational chart the evaluators needed to put the Standing Committee above the Management Board, the Director and Experts’ Forum.

Effectiveness

The Steering Committee stressed the importance of questions on communication in the evaluation, e.g. how the disseminated information is used, what happens when research findings are distributed. It should however be taken into account that not all of EIGE’s activities can be communicated, although they have their own added value.

According to the Steering Committee, in the question 2.2. the concept of“awareness” needed to be changedto “knowledge” or “better information”, while also carefully tailoring the wording of the question to different groups of informants.

The evaluation team’s idea to provide a social network analysis of EIGE was met with a concern that such a young agency cannot establish centrality. The evaluators explained that the judgments were only going to be presented where the confirming findings are available from different sources, and that in any case the purpose of the social network analysis was not to present EIGE in a negative fashion, but to provide a benchmarkwhich could enable the Institute in future to check how its centrality and density of network changes, what members it attracts, whether the communication becomes more two-sided etc.

Relevance

According to the Steering Committee, while analysing the relevance of EIGE the evaluatorsneeded first of all to look intowhetherEIGE has been able to respond to andshift the needs of stakeholders.Attention also needed to be paid to the rigidity of EIGE’s working programme, its capacity to take into account the ad hoc requests and the potential to make the work programme more flexible.

The Steering Committee also mentioned that the capacity of EIGE to react quickly to rising requests by providing at least preliminary data on the issue at hand as soon as possible needed to be analysed.

The Steering Committee suggested to change wording in question 1.7. and replace the word “financing” with “resources”.

According to the Steering Committee, the question of EIGE’s resources needed to address first of all their allocation so that EIGE could respond to the most pressing issues. The diversity of resources and possibility to adjust when needed also had to be discussed. Additionally, it was important to look into how EIGE was keeping itself updated on the trends of its policy area and the related wider field of research.

Coordination and coherence

The comment of the Steering Committee was that while doing evaluation and interviewing stakeholders, the evaluator should keep in mind that the European context was the most important for EIGEand therefore concentrate on EIGE’s European partners.

The questions 14.3. and 14.4. should reflect an issue of EIGE’s proactiveefforts to use the outcomes of other organisations.

Added value

The Steering Committee emphasized that evaluation should duly consider the added value of EIGE activities. The evaluation should look at how EIGE can better complement activities of other stakeholders. It should assess if in planning and delivering its tasks, EIGE systematically considers a broader context of its activities, if any new initiative is preceded by an in-depth ”market” analysis (availability of studies, needs of stakeholders), if it is sufficiently proactive in seeking input and confronting ideas with relevant external stakeholders (e.g. EUROSTAT, other agencies). Furthermore, it should assess the added value of contracting studies by EIGE compared to Members States contracting studies and compared to studies delivered by independent researchers.