Euro-Mediterranean Statistical Co-operation Programme
Contract: ENPI/2010/234-479
Report on the Sub-Regional Training Course on Developing and Using Questionnaires for Agricultural Surveys
Subject / Training Course on Developing and Using Questionnaires for Agricultural Surveys. Istanbul.Country / Egypt, Israel, Jordan, Occupied Palestinian Territory.
Iraq and Yemen (under funding by UNESCWA)
Dates/period / July 18-21, 2011
Document’s identity:
Author / Dr Hassan Serghini, Key expert MEDSTAT III and Mr. Marco Ballin, Short-term expert MEDSTAT III / Date / 06.12.2011Recipient(s) / DEVCO, Eurostat / Version / V3
MPCs participants in Istanbul:
Egypt: Mrs. Inas Mostafa Mohamed Hasan, Central Agency for Public
Mobilization and Statistics;
Mr. Tarek Mahmoud Mohamed Abd El Latif, Ministry of Agriculture;
Israel: Mrs. Shamir Chana Anat, Central Bureau of Statistics,
Mrs. Inbar Grinstein Decker, Ministry of Agriculture;
Jordan: Mr. Fuad Ahmad, Department of Statistics,
Mr. Falah Ibrahim Salah Hbeisshan, Ministry of Agriculture;
OPT[1]: Mr. Borhan Issa, Central Bureau of Statistics;
Mr. Iyad Ghanim, Ministry of Agriculture;
Iraq: Mr. Hilmi Thanaa, Statistical Office (COSIT) (funded by UNESCWA)
Yemen: Mr. Jammal Mohammed, Central Statistics Office (funded by UNESCWA)
Mr. Mohammed Abdel Moghini, Ministry of Agriculture (funded by UNESCWA)
UNESCWA: Mr. Hammoud Wassim, Statistics Division (own funding).
1. Introduction
The sub-regional training course on Developing and Using Questionnaires for Agricultural Surveys was held in Istanbul from 18 to 21 of July 2011, according to the work programme approved by the Task Force.
Besides participants from four MEDSTAT countries (Egypt, Israel, Jordan and Occupied Palestinian Territory), participants from Iraq and Yemen participated in this course under complementary funding by UNESCWA. Due to the difficult political situation, Syria was unable to participate. UNESCWA funded the participants from Iraq and Yemen and made a presentation on agriculture statistics in ESCWA region.
The objective of the training course was to present and discuss the well established methodologies useful to design a new or to improve an existing questionnaire for surveys carried out by Mediterranean Partner Countries (MPCs) in the agricultural sector. The training course aimed at increasing awareness about potentials problems caused by a bad questionnaire and at showing how some problems can be overcome by use of a cognitive approach.
The training course provided the opportunity for the MPCs to present their experiences and approaches on this topic, discuss common problems and difficulties and collaborate for the improvement of their questionnaires.
The participation of UNESCWA increased the synergy with this international organization, as sought by the Directors Committee of MEDSTAT.
2. Presentations on Quality concepts and quality components
The Conceptual Framework for Questionnaire Design
The KE Dr. Serghini made a presentation on this topic. The output of the conceptual frame is a precise description of the survey concepts (conceptualization) and a list of indicators and variables and questionnaire scheme (operationalization).
The conceptual frame requires intensive co-operation with users, subject matter specialists, the questionnaire designer and respondents. He reviewed the component of the framework, i.e. literature search, interplay of survey design and questionnaire design, survey objectives, conceptualization and operationalization, exploring concepts, variables and tabulation and data collection mode.
Key items emerged from discussion:
· The overview of the questionnaire design is useful for understanding the efforts needed to formulate questionnaires for agricultural surveys.
· The formulation is a process that involves all stakeholders and building up knowledge and experiences.
The cognitive model in answering questions
The STE Mr. Marco Ballin (ISTAT) presented the general concepts of the cognitive science as a tool suitable to study of the human mind as “information processors”. It has been shown that the cognitive approach can help avoiding some measurement errors that may result from:
ü respondents’ misunderstanding of the survey questions or key concepts
ü not knowing or not recalling the needed information from memory
ü from using an inappropriate heuristic for making a judgment
ü from preferring to hide certain information, or
ü from providing a socially desirable answer.
Key items emerged from discussion:
· Even if in the technical literature studies can be easily find out showing the efficiency of the cognitive approach, the discussion highlighted that it is seldom fully applied in elaborating the questionnaire for surveys carried out by National Statistical Offices;
· The literature is lacking of examples of applications on questionnaire for surveys on holdings or on agricultural topics;
· It was agreed that errors can arise in each phase of the data collection: comprehension of the question, retrieval from memory of relevant information, decision or judgment or estimation process, response or reporting process, etc.
· The comprehension of the question by farmers is considered as one of the main difficulties that the questionnaire planner faces because of technical terms that should be used. Such terms are not always understood by some farmers or can have different meaning in different regions of the countries.
· Participants agreed that memory can represent a difficulty too, because for some surveys, information about the entire year has to be collected
Principles of writing questions, type of questions and question phrasing
Since the ways the questions are asked have a major impact on behavior of respondent and, consequently, on data quality, a session has been dedicated to the principles that should be followed in writing the questions:
· the researcher should be concerned on providing questions that globally contribute to minimize errors due to the questionnaire;
· the questionnaire should be friendly both for respondent and interviewer;
· the respondent should:
◦ clearly understand what he/she is being asked;
◦ be able to answer the question and
◦ Understand how the answer has to be given.
Classification of the questions is based on the type of information collected:
· factual questions (concerning based on factual information);
· behavioral questions (questions requiring information about activities of respondent or his/her business);
· opinion questions (that is questions seeking to measure subjective opinions rather than facts);
· Hypothetical questions (that is questions on the form of “what would you do…”
· open-ended questions;
· Close-ended questions.
To limit the bias due to difficulties of respondents, some assumptions about respondents that questionnaire designer should take into account have been recalled. These assumptions about respondents are:
• They might have a limited vocabulary;
• They might understand only short sentences and can be confused by long explanations;
• They might understand positive instruction more easily than negative ones;
• They might know nothing about National Statistical Institutes’ procedures or structure;
• They might not understand why the information asked is needed;
• They might not understand the definitions or the explanations used and supplied by the NSI.
Key items emerged from discussion:
• The discussion concerned mainly the type of questions used in the case of agricultural surveys. Participants agreed that the factual questions are the most commonly used in the agricultural surveys. Experiences about embarrassing, sensitive and behavioral questions in agriculture have been illustrated by participants.
• The minimal assumptions described in the presentation seem to hold in the case of agriculture surveys;
• Participants said that quite often it is necessary to include complex questions in the questionnaire;
• To avoid some of the potential problems, participants suggested including in the questionnaire, clear and synthetic instructions as close as possible to the questions and repeating them during the questionnaire, if they concern questions in different parts of the questionnaire. Other difficulties suggested during the discussion are those linked to threatening and vague questions.
Inappropriate questions
Mr. Serghini presented the inappropriate questions as questions that are not relevant to the survey goals, unbalanced line of inquiry (questions that could be seen as developing a line of inquiry to support a particular position or preconceived idea), questions that cannot or will not be answered accurately, questions that are not geared to respondent’s depth and range of information, knowledge, and perceptions, questions that respondents may perceive as illogical or unnecessary, questions that require unreasonable effort to answer, threatening or embarrassing questions, vague or ambiguous questions and unfair questions. As groups of respondents are not all equal, the burden of inappropriate questions is different from one group to another. Therefore for avoiding them, the questionnaire designer should learn about the respondent groups, design field test for each group, and do not rely on preconceptions or stereotypes.
Key items emerged from discussion:
Avoiding inappropriate question is one of the key elements for the success of the survey. If all steps of the formulation of the questionnaire are correctly implemented, in particular the definition of clear and detailed objectives (tabulation before the formulation of the questionnaire), then many inappropriate questions can be avoided.
Questionnaire flow
The questionnaire flow, that is the sequence of the questions and topics along the questionnaire, is one of the points illustrated during the training course. The following objectives and principles should guide the questionnaire designer in planning the questionnaire flow:
• initially provide and later maintain respondents’ motivation to complete the questionnaire;
• aid respondents’ recall;
• direct respondents to the information sources they should use;
• be relevant to respondents’ own records;
• appear sensible to respondents.
Key items emerged from discussion:
The discussion focused on the properties that should have a questionnaire with respect to its flow. The cues belonging to the discussions could be summarized as follow. Questions should:
• be logical to respondents;
• move smoothly from one topic to another;
• general to specific;
• impersonal to personal;
• easy to difficult;
• don’t start with awkward or embarrassing questions;
• go from factual to abstract questions;
• go from closed to open questions;
• group the questions by similar topics;
• don’t abuse with parts or sections;
• length should be consistent with the collection mode (face to face interview can be longer then CATI interview).
An analysis of flow of the Italian Farm Structure Survey questionnaire (which is a face to face interview survey) has been carried out; participants highlighted some possible improvements concerning some personal or complex questions. It was agreed that some problems of questionnaires flow arise because of typographical constraints (layout of the questionnaire).
Response categories
Dr Serghini made clear the differences between type of questions, questions formats and response categories. , The questionnaire designer should understand the establishment of a good response list from the respondent’s perspective. The response list should contain all the categories perceived by respondents as significant to the question topics and they must not overlap. The categories must be relevant and appropriate from the respondent’s perspective. The list should not exceed 9 items. Items should be clearly defined. The order of the items in the list is important. The items should be in meaningful order.
Key items emerged from discussion:
Agricultural statistics use questionnaires with many questions with response categories. One of the difficulties is to have the same conception of the items between responds and questionnaires designers. The precise definitions of the items are rarely indicated in the questionnaire.
Visual design
Dr Serghini insisted in the importance of the visual design for the success of the survey. There are 3 levels of visual design, the “emotional level” (across cultures: immediate automatic same positive or negative reaction to a questionnaire without reasoning), the “functional level” (usability of the questionnaire, i.e. whether the information is cognitively processed by the respondent as intended by the survey designer) and the “reflective level” (the respondents take a conscious and reflected decision). Some rules for good visual design have been presented.
Key items emerged from discussion:
It is important that instructions are put as close as possible to the related questions. The visual design should be developed involving professional expertise. The visual design of a questionnaire has to be developed with respect to the main users of the questionnaire and the data collection mode. Questionnaires should be tested for usability and for their emotional level responses.
Question and Respondent Bias, memory effect and measurement error
The concept of “cognitive bias” has been introduced. Cognitive bias is a general term used to describe many distortions in the human mind that are difficult to eliminate and that can lead to perceptual distortion, inaccurate judgment, or illogical interpretation.
A classification of possible cognitive biases has been given:
• anchoring - the common human tendency to rely too heavily, or "anchor," on one trait or piece of information when making decisions;
• framing effect – drawing different conclusions from the same information, depending on how that information is presented;
• negativity bias – the tendency to pay more attention and give more weight to negative than positive experiences or other kinds of information;
• status quo bias – the tendency to like things to stay relatively the same;
• availability heuristic – estimating what is more likely by what is more available in memory, which is biased toward vivid, unusual, or emotionally charged examples;
• halo effect–the perception of some characteristics is influenced by the perception of another characteristic (an example would be the perception of the taste of a plate can be influenced by the aspect);
• social desirability bias is the tendency of respondents to reply in a manner that will be viewed favorably by others;
• telescoping effect– the effect that recent events appear to have occurred more remotely and remote events appear to have occurred more recently;
• rosy retrospection – the tendency to rate past events more positively than they had actually rated them when the event occurred;
• forgetting effect.
Key items emerged from discussion:
The discussion focused on the effects of cognitive biases when ask concrete questions about:
• total irrigated area;
• number of pesticides treatments.
Participants agreed that cognitive biases have greater effects in the case of pesticides treatments and, generally speaking, on topic where a great amount of details should be reminded.
Furthermore “rosy retrospection” and “negativity bias” are a quite common effects met in the participant’s experiences.