measuring for decision making
Official statistics for decision making and evaluation: territorial indicators
LUIGI Biggeri[1]
1.Foreward
Nobody has ever doubted that statistical information and statistical methods are indispensable for taking rational decisions. However for a long time even policy makers have rarely and badly used the statistical information available and, at the same time, producers of official statistics have not always supplied the necessary statistical data, often requested in a not well organized and articulated information demand.
Recently things have changed and governments at different levels are more and more interested in formalizing their decision processes and in evaluating their programmes, activities and especially intervention policies in economic and social areas. There are several reasons for such a renewed interest. One of them is the continuous process of decisional and institutional decentralization, with a particular attention to territory. Obviously, this implies adequate information and, above all, specific statistical information systemsand indicator sets that official statistics should implement, in the interest not only of public decision-makers but also of citizens, so that they can exercize a documented democratic control.
Istat, as the major expression of official statistics in Italy, is ever more often called on to play a considerable role in broadening statistical knowledge of social and economic phenomena for decision-making and evaluation purposes at national level. Special emphasis should be placed on deepening territorial dimension, which in Italy is characterized by highly articulated elements and significant gaps in development.
It should be noted that today, as never before, the sinergies in progress have been so intense between those who, for institutional reasons, demand for qualified information to orient, direct, monitor and evaluate policy-making and those who provide statistical information, in particular official statistics. In this overall framework, the active push coming from institutional bodies interested in a greater and more detailed statistical information helps us better investigate the fields they operate in and, above all, acts as a driving force to strengthening the vicious circle “new demand–new supply”. On the other hand, Istat has always positively reacted to such a push and has more and more often started significant experiences of institutional cooperation in the respect of the principles of reliability, transparency and impartiality required by public statistics.
In this report, a statistical theoretical approach as well as the current policy-makers demand for statistical information will be illustrated with particular focus on the production of sectoral and territorial statistics. In this context, the experience made in the construction of territorial indicators within the Operative Project will be presented. The title of the project is “Territorial and sectoral statistical information for 2001-2008 structural policies”, and it is carried out by Istat in collaboration with the Department for development and cohesion policies (DPS) of the Ministry of Economy and Finance. The aim is to highlight how information demand arises, how it is defined and specified, which are the answers as regards the construction of a set of indicators, which are the advantages and which problems are met both in the construction of the indicator system and in their interpretation. To give a reference framework and a logic order to the presentation, we will start recalling the steps that must be implemented to define, carry out and evaluate intervention policies; whereas the development and the principal uses of indicators for the development of regional policies are illustrated by Fabrizio Barca, director of DPS.
2.Policy definition and evaluation: a statistical theoretical approach
For the reasons mentioned above, the design of policy interventions and their evaluation have become, as it is used to say, a strategic activity at any level since it is an irremissible means:
(i)Of Scientific support for decisional processes to make more adequate choices and to rationalize interventions, programmes and actions.
(ii)Of Control of the feasibility and effective implementation of the programmed objectives with interventions and activities, in terms of quantity, costs, efficiency, efficacy, process and product quality and impact of implemented programmes;
(iii)For ensuring a guarantee function with reference to users, to the investments made and to the society as a whole.
To specify how policy intervention design and the ensuing impact evaluation should be organized, we can refer to the simplified framework illustrated in Picture 1 (Biggeri, 2000):
Framework analysis highlights some important and specific aspects that must be taken into consideration to organize policy design and evaluation. As a matter of fact, it is necessary:
(a)To analyze the context and real situation (a good knowledge is required of how the phenomenon works and how the involved units behave), and the problem that we have to face, to correctly plan the policy, the intervention and the connected information system;
(b)To simulate the actions on which intervention is based with macro or micro models, in order to highlight and evaluate their possibile consequences, the possibile need of modifing the intervention and, in any case, to choose among the various alternative actions;
(c)To evaluate each phase of implementation of the actions and the obtained results;
(d)To use result and evaluation analyses for learning aims (that within policy evaluation is a very important aspect) and, if necessary, to change the plan and the operative characteristics of the intervention and/or to plan or to improve the information system already available;
(e)To disseminate evaluation results also as a means of social control by general public and by interested bodies.
Pict. 1. Simplified framework of the design, the implementation and the evaluation of an intervention policy
This kind of research is different from others both because of its aims, that are precise and definite, and because of its operative conditions (that involve decisions). Both these elements influence the theoretical design of the research and its practical implementation that, as usual, will have to find an appropriate “compromise” among the three criteria of representativeness, randomization and realism (the latter in particular for explanatory variable choice).
Fundamentally for these reasons, even considering the experimental approach, the statistical design that better suits the evaluation process is the one called “controlled investigations” design that comprises the almost-experimental designs and the designs effective for observational studies or, in particular, to make comparisons; all these designs often represent ingenious adaptations of experimental designs (obviously releasing some of the criteria and of the hypotheses that are at their basis).
As everyone knows, to project these statistical designs it is essential to deeply know the nature and characteristics of the programme or of the process working model, etc that must be evaluated together with its objectives. For example, it is necessary to know which are the elements that influence programme results and how these elements presumably interact among themselves. Obviously the effects of the different characteristics of the programme and/or of the various elements must be measured usingresponsevariables strictly connected to the objectives. Undoubtedly, the analysis must be done considering the real operative conditions and consequently the characteristics of the decisional process (as highlighted in pict. 1).
It is therefore strategic to define a reference framework of the situation which is the subject of the study and/or a working model, which can be more or less detailed depending on the evaluation aims, the field of interest of the programme and considering the existing economical and sociological theories. If these theories do not exist or are not convincing, it is necessary to use empirical evidence, through a sociological, economic, managerial, engineering study of the organizations and the processes involved in the programme or in the intervention (in this case, the importance of the interdisciplinary cooperation is evident).
Two further considerations derive from the framework.
Firstly, it is necessary to have adequate quantitative and qualitative information presented, if necessary, in an appropriate and pertinent statistical information system for definition and management (of the intervention, etc) and for evaluation; there is no doubt that to plan and evaluate in an objective way, it is necessary first “to measure”, then to have adequate measures of the variables of interest and plan the opportune quantitative and qualitative indicators. It is therefore necessary to deeply know the characteristics of data and indicators and also their possible interpretative limitations (expecially if we use proxies of the theoretical indicator, without taking for granted that the supposed associative link between the proxy and the related indicator is unchangeable both in time and space); considering that generally many elements act interdependently to cause results, and it is therefore difficult to isolate the effects caused by each element; taking into account that often there is the need to“standardize” the indicators, to eliminate the effect of the structural characteristics of inputs and/or of the reference framework.
Secondly, for interpretation purposes, we always have to remember that evaluation aims at understanding if the policy (the intervention) implemented has reached its planned objective, modifing the phenomenon or, rather, the conditions and/or the behaviours of individuals belonging to the reference population observed in the real situation. As various authors underline, the fundamental methodological problem of an impact evaluation is that generally it cannot be reduced to measure only changes of a phenomenon or of behaviours between the previous situation and the situation after the intervention. Unless the coeteribus paribus condition is respected, this difference is not only attributable to the implemented policy or to the realized intervention, but it also depends on a variety of elements, among which the typical development of the phenomenon, apart from the intervention itself. What we have to measure is the net contribution of the intervention, in other words the difference between what happened after the implemented policy (factual event) and what would have happened without that policy (counterfactual event).
3.Policy-makers’ demand for statistical information: Istat efforts to satisfy the requests
As already mentioned, after a long period characterized by a substantial lack of quantitative and in particular evaluation culture, in Italy a new approach is now being asserted, more oriented towards the measurement of phenomena and its use for correct policy-making.
Istat has tried and tries to meet this demand by fostering the overall growth of the National Statistical System (SISTAN) through the National Statistical Programme and the work of sectoral “quality groups”. In addition, Istat[2] organises every year the National Statistical Conference, a significant moment where producers and users of official statistical information convene to discuss as well as to share methods, experiences and perspectives for a more effective meeting between demand and supply of public statistics. Moreover for over ten years Istat has drafted an Annual Report that is an occasion for an evidence-based reflection on Italy’s economic and social situation and for a comparison with other countries’ situations and trends.
The Report does not only deal with Italy’s macroeconomic evolution but it also contains detailed analyses on business competitiveness, labour market effectiveness and productivity, features of the welfare system and so on. The Report preparation is also an opportunity to check the quality and consistency of different data on the same phenomenon as well as data on different phenomena arising from different statistical sources. In the end, the Report highlights the most important results obtained from statistical surveys (thus increasing the value added in the presentation of the results of each single survey) and provides possible solutions to problems of economic and social policy.
Today, however, to meet policy-makers’ demand for statistical information Istat more and more often participates in ad-hoc conventions with Ministries, Regionsetc, that envisage the implementation of statistical information systems for decision-making and for evaluating the impact that action policies have on specific fields.
However, the context shows the consequences of some important shortcomings. First of all, little importance is given to the planning of information needs that will have to be met subsequently. This is especially true when planning complex evaluation actions, which often suffer from an overlapping of various disciplinary approaches (economic, sociological, political, financial etc.). Secondly, there is a tendency to underestimate the financial resources and the technical-scientific competence necessary to carry out the planned process of measurement/evaluation; furthermore it is on territory that the most important - as well as specific – information needs are expressed even if financial investments and human resources are inadequate in this field.
However, there are also cases of conventions and cooperation relationship that are gaining excellent results, both in the production of regular analysis and evaluation reports and in benchmarking exercises. No doubt, one of them is the cooperation relationship with the DPS of the Ministry for Economy and Finance aimed at building up indicators that can be used for the development of Regional Policies.
This kind of activity was requested in 1989 when regional policies had become a community goal, thus emphasising the need for all the actors involved in the implementation and programming processes to be provided with a harmonised statistical information basis. Later, this activity widely developed in the context of the problems connected to measurement and evaluation in coincidence with the start of 2000-2006 cycle of structural funds[3], both to identify areas eligible for funding and to check the criteria for breaking down financial resources.
The main results are contained in the "Programme of Southern Italy Development" (PSM), where the European pushes to the programming of development policies have been implementedin Italy in an original way, through a marked theoretical and quantitative approach in relation to the past. In this document there is a macroeconomic model containing the main instrument-objective relations through which to develop a new policy of public investments based on:
- renunciation of past welfarism
- enhancement of the existing strengths
- a system of indicators for the measurement of the planned objectives
- the relationbetween the impact of priority actions and the overall objective
Moreover an attempt was made to clarify the link between the choices made in regional and national programmes (POR) and the overall objectives established for the entire economy of Southern Italy.
The first experiences have then enabled us to launch a new phase of activities and studies aimed at supporting official statistics for decision-making through a more organic programme for the production of statistics, indicators and adequate technical tools. This has led to the already mentioned project "Territorial and Sectoral Statistical Information for the 2001-2008 structural policies” whose basic characteristics will be presented below in this report.
Such a project, that develops in a wide time lag (September 2001-December 2008) long enough to fulfil the objectives set, represents only one of the numerous experiences that Istat has been carrying out in this field. But perhaps the most representative characteristics of this project are its being an in-depth study of statistical information on territory and its implementation according to the logical framework of Pict. 1, as illustrated above.
4.The project “Territorial and sectorial indicators for structural policies”
The project mentioned above is co-financed by Istat (30%) and Structural Funds for Objective 1 areas (70%) and aims to contribute to the overall objective of increasing and improving statistical information, especially at territorial level, so that the effects of structural programmes can be better measured.
Although this project represents one of Istat’s strategic activities in partnership, it is not the only one. During the last few years many other relevant and qualified activities have been settled such as designing the information system on Handicap and Child labour within the social policy. Furthermore it has to be mentioned how relevant the surveys on Health and Social Services supply at territorial level is, which will allow the development of other two information subsystems.
In the case of the project on Territorial and Sectoral Statistical Information, the overall objectives of action policies are set at the community level and then adjusted to more specific national objectives. Wider actions at territorial level have emphasised the need for enhancing information and cognitive bases. The increased demand for statistical information arises - above all - from the need to complete and implement the set of “key context” regional indicators and the “breach variables” indicated in the Community Framework of Support for Objective 1 Regions (QCS). These are the primary tools to measure the initial socio-economic situation and the existing structures as well as to monitor progress and results of public interventions towards local development. It is therefore obvious that the main focus of this activity was - and is - what statistical information and what set of indicators are adequate for this kind of measurement and analysis with the aim of continuously improving them. It was thus necessary to study in-depth the methodological design underlying regional indicators for decision-making and evaluation.
Moreover it should be noted that each action of the project focuses on territorial variables and indicators that have been defined in close cooperation with the Department for development and cohesion policies (DPS):
a)Elaboration - for the purpose of programming development policies - of a system of regional economic accounts concerning employment within specific territorial areas (territorial sections, regions, labour local systems), even through the design of estimation models and the connection between preliminary and final series;
b)Updating, control, improvement and implementation of “key context regional indicators” and of “breach variables” since they represent the system of indicators that are at the basis of a process of macro evaluation of the effects of the actions carried out through the Structural Funds;