Euro-Mediterranean Statistical Co-operation Programme
Contract: ENPI/2010/234-479
Report on the Workshops on Improving Data Quality in Agriculture Statistics
Subject / Workshop Improving Data Quality in Agriculture StatisticsSessions for French and English speaking countries
(Casablanca, 16-19 May and Brussels, 30 May-2 June 2011)
Country / All MPCs
Dates/period / May 2011-July 2011
Document’s identity:
Authors / Dr Hassan Serghini, Key expert MEDSTAT III and Mr Valerio Lucchesi, Short-term expert MEDSTAT III / Date / 20.06.2011Recipient(s) / DEVCO, Eurostat
MPCs participants in Casablanca:
Algeria: Mrs Fatiha Ghemmaz, National Statistical Office;
Mr Mouloud Lounis, Ministry of Agriculture;
Lebanon: Mrs Mayssaa Daher, Administration Centrale de la Statistique;
Mrs Rima El Hajjar, Ministry of Agriculture (whose participation was covered by her institution);
Mr Hammoud Abou Diab, Ministry of Agriculture;
Morocco: Mrs Souad Chaoui, High Commission for Planning (HCP)/ National Accounts Division;
Mr Mostafa Tahri, Ministry of Agriculture;
Tunisia: Mr Habib Zouhaier, National Statistical Institute;
Mr Abdallah Medfai, Ministry of Agriculture and Environment;
MPCs participants in Brussels:
Egypt: Mrs Samia Fahmy Amin Keleny, Central Agency for Public Mobilization and Statistics;
Mr Mohsen Elbatran, Ministry of Agriculture;
Israel: Mrs Hana Tubi, Central Bureau of Statistics,
Mr Jamal Madlege, Ministry of Agriculture;
Jordan: Mr Bassam Zain, Department of Statistics,
Mr Ebraheem Khaltab, Ministry of Agriculture;
OPT[1]: Mrs Shadia Abu Alzain, Central Bureau of Statistics;
Mr Basel Amro, Ministry of Agriculture;
1. Introduction
Two sessions of the workshop “Improving Data Quality in Agriculture Statistics” took place, the first was held in Casablanca from 16th to 19th of May, in French, for Algeria, Lebanon, Morocco and Tunisia and the second in Brussels from the 30th May to the 2nd of June 2011, in English, for Egypt, Israel, Jordan and the oPt. Due to the difficult political situation Syria was unable to participate.
The objective of the workshops was to discuss and find ways to improve data quality in agricultural statistics in the Mediterranean Partner Countries (MPCs) with particular reference to European and international quality standards. The workshops aimed at increasing awareness of the multiple sources of errors in agricultural statistics, their measurement techniques as well as of the quality concepts and approaches used in EU countries. The workshops also provided the opportunity for the MPCs to present their experiences and approaches on quality, discuss common problems and difficulties and collaborate for the improvement of quality standards in the MPCs.
Mr Lounis (Algeria) and Mrs Abu Alzain (OPT) were elected chairs of the two respective workshops. In both sessions, the participants regretted the absence of Eurostat and FAO officials and the fact that not a unique session with all MPC countries had been organized as this would have allowed a broader exchange of experiences.
2. Presentations on Quality concepts and quality components
General Quality Concepts
The STE Mr Lucchesi presented the general quality concepts, how they fit into the statistical context and the needs and peculiarities of agriculture statistic framework. He stressed the need for a broader concept of statistical quality that goes beyond the traditional producer and output oriented approach, in favour of a user and process-oriented attitude. The concepts of “Total Quality Management System” and “Quality Management Framework” were also presented and discussed together with the European Statistical System (ESS) definition of quality and ESS quality policies.
Key items emerged from discussion:
· Quality of agriculture statistics is crucial for many reasons in the MPCs, first of all to monitor food security and improve sustainable management of agricultural resources.
· Problems of financial and human resources hamper the development of high standards for quality statistics in the MPCs. Policy commitments on improving statistical quality should be associated to the efforts of statisticians.
· Better coordination across different sub-domains of agriculture and non-agriculture statistics (employment, consumption...)is a need highlighted by participants to improve the quality approach.
· Independence of statistics cannot always be assured especially if agriculture statistics are produced by services of the Ministry of Agriculture (MA). The fact of having staff from National Statistical Institutes (NSIs) working in the MA was mentioned as an example of good practice.
· In the MPCs there is a strong need for better trained staff with particular attention to both the specificities of agricultural sector and quality issues.
· The concept of “Efficiency” (i.e. producing statistics at minimum cost to the NSI and to the respondents) should be more carefully considered in the MPCs and it should be explicitly associated to quality approaches and in the allocation of resources.
· An issue highlighted by participants is that the quality approach is a long-term investment whereas policy-makers are sometimes exclusively focussed on short-term matters.
Relevance
The KE in Agriculture Statistics Mr Serghini presented the main issues related to the quality component “Relevance”[2]. He highlighted the importance of relevance of the final data produced but also relevance of the used statistical concepts and definitions. Regular revisions and consultations with users (not only MA only but also farmers’ organizations, the private sector and the academia) have to be carried out. The problems related to relevance have to be specifically considered when the NSI use administrative data as the concepts might be different from those used for statistical purposes. Indications for presenting the relevance of data in a “Quality report” were also presented and discussed.
Key items emerged from discussion:
· The organization of an “agriculture statistics day” was mentioned as a good practice. This sort of events are to be held regularly (e.g. annually) to get together producers of statistics and users for specific consultations. Statistical concepts have to be discussed with users to check if they correspond to their needs. Difficulty in finding the acceptable balance between needs and available means has been highlighted.
· Egypt presented their experience on organizing such events. The MA organized a survey for users through the use of a specific questionnaire sent to almost 500 stakeholders of different nature. The analysis of the results was presented and discussed during a workshop. A publication presenting the main outcomes is available “Choose your own data” and it will be shared among the participants.
· MPCs noted that there is a need for an explicit and regularly updated list of needs from users (e.g. in the form of an annual document) with also reference to the required periodicity.
· The quality component of relevance is difficult to be measured through indicators. A good practice noticed is the use of an internet survey to receive feedback on users’ satisfaction.
· Several participants regularly send the survey questionnaire to MA for comments and formal meetings are often held (in Morocco, for example meetings include also farmers’ organizations and academia for broad consultation).
Timeliness and punctuality
Mr Lucchesi presented timeliness and punctuality[3] as crucial elements of data quality for effective use of results which allow policy-makers to take informed decisions in time for achieving the targeted results. The difficulty of the compromise between timeliness / punctuality and accuracy and relevance has been discussed. Methods to measure timeliness and punctuality were presented including the example of the detailed evaluation of the punctuality of survey phases made by ISTAT (Italy) in its “Farm Structure and Production quality report”. Reference was made to several EU regulations including official calendars and to EU good practices (i.e. from UK, France and Portugal).
Key items emerged from discussion:
· A discussed strategy for making data timelier is when agriculture statistics are released in several versions (preliminary, revised and final) but the right trade-off with data accuracy is not always straightforward to be identified.
· MPCs use calendar for data releases, these are established by NSI and are not reported in official legislations.
· Cultural differences due to the interpretation of the concept of “punctuality” in the MPCs were mentioned. A systematic measurement of this quality component is generally not in use in MPCs, calendars do not systematically exist and, not always, are publicized and respected.
· The value of the use of Information and Communication Technology (ICT) to improve timeliness and punctuality was mentioned.
Accessibility and clarity
Mr Lucchesi presented concepts and definitions for both accessibility and clarity[4] and highlighted the need to refer to different kind of users. The use of modern information and communication technology as well as traditional hard copy as dissemination formats was discussed. Eurostat’s good practices to allow accessibility and clarity of data were presented, including the existing on-line database, the different types of agriculture and horizontal publications and the use of maps produced by Geographical Information Systems. Examples from France –as an illustration of good practices- were also shown. The importance of documentation of metadata according to standardised systems was mentioned and examples of indicators for measuring both accessibility and clarity were listed.
Key items emerged from discussion:
· Even though printed publications always exist in MPCs for main agriculture aggregates, on-line databases do not systematically exist as participants recognized that the ICT culture is not as developed as in the EU and internet access is not as widespread. Lack of financial and human resources for development of statistical gateways is an issue in the MPCs. So far only pdf documents are published on internet in most cases.
· In most MPCs agriculture databases are often accessible through intranet for official users (mainly the MA) and often have different level of users’ accessibility depending on confidentiality degree of the strategic information reported.
· In Morocco a “statistical information desk” is available at the HCP – Statistical Directorate to answer specific questions. A “Researchers’ room” exists at Central Bureau of Statistics of Israel with possibility of access to micro-data according to specific rules for data display and use.
Coherence and comparability
Mr Serghini presented concepts and definitions for both coherence and comparability[5] over time, regions and statistical domains. He highlighted the main and common reasons for lack of coherence and comparability which are normally caused by differences in concepts or methods -or both-. Methods for assessment of these components were discussed (both general approach that apply to all type of coherence/ comparability and the specific methods). Reporting indications for these quality components were presented.
Key items emerged from discussion:
· Comparing data is a useful exercise to evaluate problems and understand where problems are. Mirror statistics (e.g. in trade statistics across countries or in the balance of payments) provide useful tools for evaluation of the coherence and comparability of data. The compilation of supply balance sheets was also mentioned as an opportunity to evaluate comparability and coherence of different data sources related to the agricultural sector.
· Coherence of agriculture statistics with the National Accounts was discussed: the results of comparisons with National Account and feedback from National Accounts units provide insight to coherence -but also accuracy- problems.
· Cases of difficulties in comparing time series in the MPCs were mentioned (e.g. in Palestine, since 2010, agriculture statistics are calculated by holdings and not by localities as they previously did).
Accuracy
Mr Serghini presented the general concepts related to accuracy[6]: the estimation of accuracy is difficult exercise as it is intrinsically a multivariate problem, where each error source contributes to the systematic and variable error of the estimate. The characteristics of the “Quality profiles[7]” were discussed as tools to provide qualitative and quantitative information about total survey error and its principal components. It has been stressed that the breakdown of all survey errors are of great importance for identifying areas that most need improvement and relative importance of each error areas (e.g. sampling vs. measurement error).
Mr Lucchesi presented the issues related to sampling errors highlighting that a detailed presentation of random errors is usually best made when dealing with sampling techniques in a quality report. Several presentational devices of sampling errors were mentioned such as standard errors, coefficients of variation (CV) and confidence intervals. Reference was made to thresholds included in EU regulations related to agriculture statistics and some examples of sampling errors treatment in EU countries quality reports were presented.
Non-response errors were also subject of a presentation including discussion of the factors associated with both unit and item non response and their consequences on quality of data. Suggestions -based on EU experiences- for measurement and description in quality reports of these types of errors were also presented.
Coverage errors were subsequently discussed in detail, including issues related to over-coverage, under-coverage, misclassification and suggestions for relative indicators. It was stressed the importance of the availability of an updated farm register as a crucial element of a good infrastructure for agricultural statistics; the “Post-census coverage and quality survey” carried out in Italy and the example of the farm register in Austria were discussed.
Mr Lucchesi also presented the main issues related to the treatment of processing errors affecting both micro and macro-data. Different sort of errors were defined such as data entry, coding, editing, imputation, weighting and classification errors. Examples of indicators and description of these kinds of errors in EU member states were presented for discussion.
To conclude the session of the workshop dedicated to the component “Accuracy” of statistical quality Mr Serghini presented the issued related to measurement errors due to the effects of design and contents of the questionnaires, the effects of data collection methods, those of the interviewers on the responses and the effects of the respondents. The common approaches used to quantify measurement error were also shared with the participants.
To conclude the session of the workshop dedicated to the statistical output components, Mr Serghini highlighted in a presentation the main aspects related to the trade-offs between the different quality components.
Key items emerged from discussion:
· Participants agreed that “accuracy” is the most frequently and traditionally considered component of quality (sampling errors in particular).
· Coefficients of Variation (CV) or other indicators of sampling errors are normally calculated in the MPCs but they are not systematically published. It has been recognized that in case of complex sampling techniques the calculation of CVs is not always straightforward.