Guidelines for Creating Questionnaires

Guidelines for creating Questionnaires

Introduction

It is virtually impossible to give a set of rigid rules for setting up a questionnaire. This document therefore provides only general guidelines.

Planning prior to setting up a questionnaire

• The purpose of the investigation or research project must be established before the questionnaire is drawn up. The purpose dictates what type of information should be obtained with the questionnaire. At this stage the researcher must do a thorough literature study on the subject under investigation.
• Decide whether a questionnaire is the most suitable method to capture the desired information.
• The purpose of the investigation will determine the target group, that is the respondents to whom the questionnaire is directed.
• The target group will determine the manner in which the questionnaire is to be completed. In other words whether the questionnaires will be posted to the respondents, completed in a group under supervision, be completed during personal interviews, etc.
• Consider the conditions under which the questionnaire is to be completed. The conditions in a hut in the informal sector differ considerably from conditions in an air-conditioned hall with comfortable desks.
• Discuss the size of the sample, the definition of the population and the actual drawing of the sample with a statistician. The final draft of the questionnaire must also be discussed with the statistician and the research consultant. The hypotheses to be tested must preferably be formulated beforehand to ensure that all relevant information is included in the questionnaire.
• Ensure that all information which may later be necessary for evaluation of the realisation of the sample is included in the questionnaire.
• The question of anonymity must be addressed. It must be clearly explained to the respondent that all information will be considered confidential and that no personal information will be published. No information that can identify the respondent should be on the questionnaire. For example, the questionnaires should not be numbered prior to sending it to the respondent. An exception must be made in the case of studies where information must be obtained before and after certain events or actions and the two sets of data must be linked. In this case information such as a student number must be used and the situation must be explained to the respondent.
• Do a pilot study to test the questionnaire. If this is not possible the questionnaire must be tested on a few persons similar to persons in the target group. This will help to eliminate possible inherent problems.

General remarks regarding the contents

• The target group determines the level of the questions. The contents of the questions as well as the language and the terminology should not overestimate nor underestimate the abilities of the respondent.
• The length of the questionnaire is determined by the size of the sample as well as by the target group. A very long questionnaire frustrates and exhausts the respondent. If the sample size is not sufficiently large, questions with too many categories as possible answers can lead to frequencies that are so low and "sparse" that the information can either not be used at all or the categories must be grouped together to be of any use at all.
• The language must be correct and unambiguous. It is important that the possible answers can be linked to the question. Questions must be restricted to one topic only.

• Psychometric questionnaires, that is questionnaires measuring aptitude, IQ etc are not discussed here. Certain psychometric concepts are, however, relevant to the development of questionnaires.
• Construct validity indicates whether relevant constructs or concepts are being investigated. For example, in a questionnaire about agricultural problems, valid constructs would be droughts, maintenance of implements, etc
• Content validity concerns the contents of the questions measuring the constructs. If the construct is, for example, maintenance of implements then "quantity of lubricants bought" is valid, but "amount spent on dipping fluid" is not.
• Reliability indicates the extent to which the questionnaire will give the same results repeatedly. Say, for example, the question is "How much did it rain?" The possible answers "Plenty" and "Not much" will not have the same meaning in wet years as in dry years. On the other hand "More than 40mm", "30-40mm" and "Less than 30mm" will have a constant meaning.

A few concepts

• Variablese.g.Gender, home language, denomination, distance
• Associated with each variable is one of the following:
• Categories e.g.Male, Female, Afrikaans, English, Tswana
• Values e.g.10, 15, 20km or 5, 10, 15 years
• Scale values e.g.Excellent, Good, Moderate, Poor
• Scale types
• Nominal e.g.Male, Female
• Ordinal e.g.Excellent, Good, Moderate, Poor or the five classes of a Likert Scale
• Interval/Ratio e.g.Temperature, Mass, Distance, Age in year intervals

Classification of question types

Questions can be classified according to their function, which in turn isdetermined by the purpose of the questionnaire. The classification in the following table is a guideline. It is, however, important to remember that it seldom happens that only one kind of question appears in a questionnaire.

Purpose / Type of question
Description / Biographic
Explaining / Behaviour and knowledge
Planning / Knowledge
Predicting / Attitude

Discussion of different types of questions

Biographical questions

Only the necessary information must be collected.

Questions requiring sensitive information from the respondent must be carefully worded.

Age must preferably be posed as an open question, and if necessary, categorised at a later stage. This enables one to use a larger variety of statistical methods.

The statistical techniques that will be used must be kept in mind during the compilation of the questions. If the researcher plans to use techniques such as regression analysis and analysis of variance then the information must, where possible, be collected in an uncategorised form.

Questions must, as said before, be unambiguous and not open to misunderstanding.

Poor example:

Do you travel by bus in the mornings or in the evenings? Yes/No

Better option:

Do you travel by bus in the mornings? Yes/No

Do you travel by bus in the evenings? Yes/No

The possible answers provided for a question must be logical and relevant for the target group. Once again, the researcher must be sure that the required information is collected.

For example, if the target group is children between 13 and 16 years of age and the question is about highest educational qualification, the following options will not give much information

None / Grade 3-7 / Grade 8-12 / B-degree / Post-graduate

Instead, the following options would be more relevant

None / Grade 3-6 / Grade 7 / Grade 8 / Grade 9 / Grade 10 / Grade 11 / Grade 12

Questions designed to determine frequency of events can be problematic. Once again, the solution is a consistent clear formulation.

Poor example:

How often do you consult a doctor?

Regularly, frequently, occasionally, never

The concept frequently will definitely not have the same meaning for everybody

Better example:

How many times did you consult a doctor between January, 1 2003 and December,31 2003?

Once, two or three times, four or five times, more then five times, never

In the last example the period is clearly defined, the frequency of visits is stated unambiguously and provision is made for respondents who did not visit the doctor.

Sifting questions

In the previous example a sifting question can be used and followed by an open question.

Example:

Did you consult a doctor during the period...... ? Yes/No

If yes, how many times? ......

The advantage, again, is that more statistical techniques can be used.

Open questions

Open questions have no formulated categories for possible answers. This type of questions are usually used when insufficient knowledge regarding the particular subject exists and the researcher is uncertain whether predefined categories will cover all possibilities. Open questions are, therefore, particularly useful for exploratory studies. It can also be used to obtain more information regarding previous closed questions or to test the intensity of the response. As the coding of this type of question is time consuming it should only be used when absolutely necessary.

"Answer each one" or "Choose only one"

The use of these two types of answers depends on the question and the purpose of the study. "Answer each one" is used when there is more than one possible answer to the question. It also forces the respondent to take each given category into consideration. The following method is recommended:

Why did you start smoking?

Wanted to know what it felt like / Yes / No
I was dared / Yes / No
I wanted to prove myself / Yes / No
I did not want to be different from my friends / Yes / No
Boredom / Yes / No
I heard that it was relaxing / Yes / No

If the respondent was asked to choose all relevant reasons, as in the next example, it cannot be assumed that a reason that has not been chosen yields "No" as in the previous example.

Why did you start smoking? Choose all relevant reasons.

Wanted to know what it felt like
I was dared
I wanted to prove myself
I did not want to be different from my friends
Boredom
I heard that it was relaxing

"Choose only one" is used in cases where the question indicates that the most important reason or aspect must be chosen.

The target group must be considered, as it is sometimes easier for respondents to answer Yes/No to each possibility than to indicate the most important reason.

Ranking

Questions that expect the respondent to rank a number of reasons, causes or anything else must only be used when the researcher is sure that the respondents will be able to manage this. Even then the number of possibilities to be ranked must be kept to a minimum. It must also be clearly indicated that the respondent must rank every possibility.

Direct against sympathetic approach

Example:

Are you overweight?

Against:

Do you think that you are overweight?

Again there can be no fixed rule. The researcher must be guided by the target group, the nature of the study and the specific question. Sometimes a more sympathetic approach or a question that is worded in a conversational style would be more effective in motivating a respondent to answer a sensitive question. The length of the question must, however, be restricted as much as possible.

Time dimension of events

Do not overestimate the respondent's memory. Questions must be restricted to important or recent events.

Unreasonable demands must not be made on the respondent. The relevant time period must be clearly defined.

Example:

Did you buy a new car during the past year?

What is regarded as the past year? 2003? April 2003 to March 2004?

It is sometimes possible, given the subject and the size of the sample, to make deductions over time by restricting the question to a specific time.

Example:

How many times did you go to church during August?

(Where August is the month preceding the survey)

If the sample size is sufficient and one assumes that the church attendance pattern is relatively constant, it is reasonable to expect that exceptional behaviour will be cancelled out and that the monthly church attendance of respondents will be reflected.

Reliability

It is difficult to check the answers to biographical questions and these questions must be formulated in such a way that the respondent can give honest answers.

Knowledge questions

Most questionnaires include such questions not to test the respondent’s knowledge as such but because knowledge is the foundation of certain behaviour or because it helps to form attitudes.

Sifting questions are valuable in this case, if a respondent indicates that he/she has never heard of a chiropractor, then it does not make any sense to ask this specific respondent any questions regarding the work of a chiropractor.

The respondent's frequency of contact with the subject must also be taken into account. The knowledge about bus routes of a person who travels by bus every day will be more "reliable" than the knowledge of a person who only travels by bus occasionally.

For the same reasons as discussed previously with the Yes/No answers, the preferred type is True/False. In some cases "Don't know" or "Uncertain" can be added as a category. The two possibilities are not equivalent and must be handled carefully.

Questions regarding opinions and attitudes

With regard to attitudes, there is a difference between determining the nature of the attitude and determining the intensity of the attitude.

In the first case the answer to each question is important and must be reported separately. In the second case a number of questions that theoretically measure the construct must be included. Techniques like factor analysis must then be used to ascertain that this is proved in practice. The group of questions that clearly measures a construct is then used as a entity and values or totals can be reported for the constructs.

The different scales that can be used will now be discussed.

• Dichotomous scale

Example:

The bus service satisfies my needs / Yes / No
Buses depart on time / Yes / No
Bus tariffs are too expensive / Yes / No

In this case the answer to each question is analysed on its own.

• Likert scales

This type of scale can be used to measure intensity of feelings. As mentioned before, a specific theoretical construct is supposed and with the help of statistical techniques it must be determined whether the data confirms the theory.

Example:

Strongly Agree / Agree / Neutral / Disagree / Strongly Disagree
Travelling by bus is time consuming / 1 / 2 / 3 / 4 / 5
Buses are always late / 1 / 2 / 3 / 4 / 5
Bus drivers drive recklessly / 1 / 2 / 3 / 4 / 5

• Diagram scale

This scale determines the respondents' tendency to both sides of a central point.

Example:

Punctuality

:: / :: / :: / :: / :: / :: / :: / :: / :: / :: / :: / ::

- 0 +

• Semantic differential scale

This is an extension of the diagram scale as the concepts that are measured in words to the scale.

Example:

BUS SERVICE

Excellent :_:_:_:_:_:_:_: Poor

Clean :_:_:_:_:_:_:_: Dirty

Quick :_:_:_:_:_:_:_: Slow

Whenever this scale is used it is important to give a proper example.

General remarks regarding scales

When deciding which statistical techniques to use it must be remembered that the distances between the possible answers do not invariably have the same value.

The number of categories that are used must be restricted to the minimum.

A problem, that may arise with certain target groups, is that respondents choose all the neutral answers or answers that are considered to be "desirable". This problem can, largely be avoided by giving proper instructions, asking relevant questions and, while coding the questionnaires, being mindful of problem cases.

The difference between knowledge, opinions and attitudes

The difference is not always very clear and the literature sometimes differentiates on a continuum between cognitive and evaluative attitudes. In general, a question with "Don't know" as a possible answer indicates measurement of knowledge, whereas a question with "Unsure" as a possibility in turn indicates measurement of attitude.

The phrasing of questions

This is the most important aspect of the questionnaire and must be properly attended to.

Example:

Speeches against democracy should not be allowed.

will elicit a different response to:

Speeches against democracy should be prohibited.

"should not be allowed" is more neutral than "should be prohibited"

Adjectives indicating intensity should be used with care.

Hypothetical questions rarely yield much information. The respondent must be familiar with the subject or object regarding which his attitude is being determined.

Where examples on how to answer questions are given, these must be clearly separated from the rest of the question and identified as an example. Examples must merely show how to answer questions and not be indicative as to the type of answer that is expected.

Sequence of questions

As a general guideline similar questions and questions regarding a common subject should as far as possible be grouped together to give the questionnaire a logical flow. In some cases it might be better to put sensitive questions, that the respondent may possibly refuse to answer, at the end of the questionnaire. In this way the maximum amount of the information will be gained from the respondent.

Source

Adapted and translated by Rina Owen from:

CS Steenekamp: "Praktiese riglyne vir vraelyskonstruksie"