A survey is a systematic quantitative method that aims to collect information from a sample of the population such that the results are representative of that population within a certain degree of error. Surveys are primarily used to collect information (through the use of a structured and standardized questionnaire) on the perceptions, opinions, knowledge, attitudes and behaviour of a sample using primarily closed-ended questions. Surveys are not be used to explore opinions in-depth.
Instructions: The steps1. Clarify the purpose of the survey
Figure out what you are trying to learn and have a clear image in your mind of the research objectives. This will help you determine whom you will survey, formulate the questionnaire and decide on the kind of statistical analysis you need to perform. If your goals are unclear, the results will probably be unclear.
2a. Select your sample
Usually, the population will be too large for you to attempt to survey all its members. A small, but carefully chosen sample can be used to represent the population. Sampling is used to cut costs and effort while still obtaining information from a representative sample of the target population. It is essential that the number of individuals participating in the survey be large enough to produce results that are reliable and valid and represent the target population with the minimum statistical error. If unsure, there are some websites that help you calculate your sample size and your error (e.g., etc)
The main questions in selecting your sampling design are (apart from your budget of course):
-Which are the demographics?
-What is the size of your target population?
-Are you interested in any sub-groups of the population? It is sometimes useful to establish quotas to ensure that your sample accurately reflects relevant sub-groups in your target population.
-How many will be included?
2b. Sampling methods
Sampling methods can be either probability (each member of the population has an equal probability of being surveyed) or non probability (members are selected from the population in some non-random way). The advantage of probability sampling is that sampling error can be calculated. In non probability sampling, the degree to which the sample differs from the population remains unknown.
Probability methods
Random sampling: the purest form of probability sampling. Each member of the population has an equal and known chance of being selected. When there are very large populations, it is often difficult or impossible to identify every member of the population, so the pool of available subjects becomes biased.
Systematic sampling: After the sample size has been calculated, every Nth record is selected from a list of population members. As long as the list does not contain any hidden order, this sampling method is as good as the random sampling method.
Stratified sampling: The population is divided into groups of individuals that are similar in some respect. After the population has been divided into the strata, a random selection of a proportion of individuals from each stratum is made.
Cluster sampling: used if the target population is spread over a large geographic area. The survey area (such as a territory) is divided into clusters. A random sample of these clusters is drawn and individuals within the cluster participate in the survey.
Non probability methods
Convenience sampling: you simply take what is “convenient”. In this type of sampling you cannot measure the degree of confidence because the group selected may not be representative of the entire population. However, sometimes representativeness is not as important as ensuring that you have specific individuals selected. This method is often used during preliminary research efforts to get a grosso modo estimate of the results, without incurring the cost or time needed to select a random sample.
Judgment sampling:you select the sample based on judgment. This is an extension of convenience sampling. For example, a researcher may decide to draw the entire sample from one representative area, even though the population includes all cities. When using this method, you must be confident that the chosen sample is truly representative of the entire population.
Quota sampling: it is the non probability equivalent of stratified sampling. Like stratified sampling, you first identify the strata and their proportions as they are represented in the population. Then convenience or judgment sampling is used to select the required number of subjects from each stratum. This differs from stratified sampling, where the stratums are filled by random sampling.
Snowball sampling is a special non probability method used when the desired sample characteristic is very rare. It may be extremely difficult to locate respondents in these situations. Snowball sampling relies on referrals from initial subjects to generate additional subjects.
3. Decide on your survey method
The methodology will affect response rate, the kind of questions that can be asked and how they are asked. When choosing the most appropriate method you must consider the best way to communicate with potential respondents, skills of staff / interviewers, the availability of internal resources and time available. The key is to choose a method that will:
-give you the highest response rate with your particular target group
-be the most convenient for them
-fit your time line
-fit your budget
-fit your staff and resources
The primary methods for obtaining survey research are the following:
a) Personal / face-to-face Interviews
Advantages
-Usually gives a higher response rate
-Preferable for addressing complex issues where some explanation may be needed or when or visual materials are shown
-Reduces non-response to individual questionnaire items
-The ability to find the target population. For example, you can find people who have attended an energy-change initiative much more easily outside the location in which it is taking place than by calling phone numbers at random.
-Longer interviews are more easily tolerated.
Disadvantages
-Social desirability bias
-More time consuming and expensive
b) Telephone interviews
Advantages
-It is possible to achieve high response rates
-The amount of non-response to questionnaire items can be minimized
-Quick results
-Less costly than face to face interviews
-You can dial random telephone numbers when you do not have the actual telephone numbers of potential respondents
Disadvantages
-Sometimes difficult to reach a selected resident of a household
-Long or complex questions should be avoided
-Can be confused with telemarketing sales, which drops the response rate
-No working person is home during the day. This limits calling time to a "window" of about 18:00-21:00.
-You cannot show visual material for assessment
c) Mail interviews (self completed)
Advantages
-Social desirability bias is minimized
-Cost effective
-The questionnaire can include pictures
-They are not intrusive
Disadvantages
-You need to ask simple, easy to answer questions, as it is self completed
-Some questions may not be complete on returned questionnaires
-Too much time is needed until all questionnaires are filled and sent back to you
-You can not be sure of who exactly has filled in the questionnaire
-You cannot reach populations of lower educational and literacy levels, or immigrants
-Low response rate. One way of improving response rates is to mail a letter telling your sample to watch for a questionnaire in the next week or two. Another is to follow up a questionnaire mailing after a couple of weeks with a card asking people to return the questionnaire. Another way is to use an incentive. Another possibility is to include the people who return completed surveys in a drawing for a prize. A third is to offer a copy of the top line results to those who complete the questionnaire.
d) Email Surveys (self completed)
Advantages
-Fast results
-Economical
-You can attach pictures and sound files
-The novelty element of an email survey often stimulates higher response levels than ordinary mail surveys.
Disadvantages
-Email surveys do not reflect the population as a whole in cases with low internet usage penetration, or practical exclusion of older target groups
-You cannot use email surveys to generalize findings to the whole population. People who have email are different from those who do not, even when matched on demographic characteristics, such as age and gender.
-You need to ask simple, easy to answer questions, as it is self completed
-You must possess (or purchase) a list of email addresses.
-Some people will respond several times or pass questionnaires along to friends to answer
-People can easily quit in the middle of a questionnaire. They are not as likely to complete a long email questionnaire as they would be if talking with a good interviewer.
-Many people dislike unsolicited email.
e) Internet (Web Page) interviews (self completed)
Advantages
-Fast results
-Economical
-You can show pictures, video and play sound
-The novelty element often stimulates higher response levels
-You can use complex question skipping logic and randomizations
-Some people will give more honest answers to questions about sensitive topics
-On average, people give longer answers to open-ended questions on internet questionnaires than they do on other kinds of self-administered surveys.
Disadvantages
-Internet surveys do not reflect the population as a whole in cases with low internet usage penetration, or practical exclusion of older target groups
-You need to ask simple, easy to answer questions, as it is self completed
-People can easily quit in the middle of a questionnaire. They are not as likely to complete a long questionnaire on the web as they would be if talking with a good interviewer.
-Depending on your software, there is often no control over people responding multiple times to bias the results.
4. Design your questionnaire
-The first step in writing a questionnaire is to determine if there is an existing questionnaire (published literature or other organizations similar to your own) that can be used to collect the information you want, even if you need to adjust it. If there are no existing questionnaires available, then it will be necessary to design a new instrument.
-The first rule is to design the questionnaire to fit the medium (i.e., telephone, internet, etc)
-Start with an introduction or welcome message.
-Ideally, the early questions in a survey should be easy and pleasant to answer. These kinds of questions encourage people to continue the survey. In telephone or personal interviews they help build rapport with the interviewer. Grouping together questions on the same topic also makes the questionnaire easier to answer.
-Avoid the temptation to ask questions because it would be "interesting to know". As a general rule, avoid long questionnaires.
-Include other experts and relevant decision-makers in the questionnaire design process.
-Use simple, comprehensive and to-the-point language. The questions must be clearly understood by the respondent.
-Allow a “Don't Know”, “Other” or “None” response to questions. In most cases, these are wasted answers as far as the results are concerned, but are necessary alternatives to avoid frustrated respondents.
-Leave your demographic questions (age, gender, income, education, etc.) until the end of the questionnaire.
Qualities of a Good Question
-Avoids emotionally charged or leading words and phrases that point towards a certain answer.
-Asks for an answer on only one dimension. Do not put two questions into one. For example if you ask “To what extent did the X initiative influence you to unplug your TV when not in use and change your energy saving settings in your PC?”, the respondent will not know to which of the two to answer. A good question asks for only one "bit" of information.
-Can accommodate all possible answers so that you do not confuse and frustrate the respondent. Add all the necessary response categories and allow multiple responses.
-If you want only one answer from each person, ensure that the options are mutually exclusive. A good question leaves no ambiguity in the mind of the respondent. There should be only one appropriate choice for the respondent to make.
-Follows comfortably from the previous question. Transitions between questions should be smooth.
Question Types
a) Open ended vs. closed questions
-In closed questions, response categories are provided and interviewers / respondents simply have to circle or choose an option
-Open-ended questions are asked without specific response alternatives; respondents need to write / say their own answer. They are good for depth and unbiased opinions
-Open-ended questions are best used when having multiple response options may be too leading and result in biased answers
b) Multiple-choice
-In such questions, respondents choose one or multiple response answers
-Make sure you include all the relevant alternatives as answer choices.
c) Scales (e.g., rating, ranking or agreement)
-Scales is an issue on which researchers differ; whether to use a scale with an odd or even number of points. Some like to force people to give an answer that is clearly positive or negative. Others feel it is important to offer a neutral, middle option.
-Ranking example: Of the following 3 services, which one would you feel is most important? Which one is second-most important? And which one is the third-most important?
-Agreement example: On a scale of 1 to 5 where 5 is strongly agree and 1 is strongly disagree how would you rate your agreement or disagreement with the following statement…
If you need help designing the questionnaire, you can download a free questionnaire wizard (e.g.,
5. Pre-test the questionnaire – conduct pilot interviews
The last step in questionnaire design is to test it with a small number of interviews before conducting your main survey. Ideally, you should test it on the same kinds of people you will include in the main study. If that is not feasible, have a few people, other than the question writer, try the questionnaire. Piloting can reveal unanticipated problems with questionnaire length, wording, comprehension, question sequence, instructions to skip questions, inadequate response alternatives in closed questions, etc. It can help you see if the interviewees understand your questions and give useful answers. After the pilots, you can revise the questionnaire, accordingly, if needed.
6. Conduct the interviews (for face-to-face and telephone)
A good interviewer follows a few simple guidelines which ensure detailed, accurate, and unbiased data. Some tips follow:
-Read the questions as written, do not change the wording
-Do not suggest responses (with verbal or non verbal cues); allow the respondent to answer without input from you.
-If respondents get off topic in open ended questions, you should remind them of the question. Sometimes repeating the question will help. If the respondent has difficulty thinking of a response, resist the temptation to “help”.
-Record answers neatly and thoroughly
-Occasionally, a respondent will want to terminate before completion. If this is the case, you should make sure the interview gets done. This can be accomplished by telling the respondent how important his/her opinions are and working together to get through the interview quickly. If it is impossible to complete the interview, arrange a time to call back at the respondent’s convenience to complete the interview.
-For telephone interviews
- Do not simply hang up when someone refuses to participate. Politely ask the respondent why they do not wish to take part. If they are busy you can arrange a more convenient time to call back. If they are concerned about the nature of the study, you can answer their questions. If they are nervous about offering their opinions, you can reassure them that all information is strictly confidential and there are no right and wrong answers.
- Try calling back 3 to 5 times before assuming you cannot reach the person at that number.
- You should record on the sample list what happens every time you attempt to do an interview with a person (refusal, absence, wrong number, ineligible respondent, scheduled call-back, busy line, etc).
-For face to face interviews
- The most appropriate location to conduct a face to face interview is a place where your sample frequents and is comfortable for them to participate. If you are randomly selecting respondents, it is important to utilise more than one location in order to ensure a better representation.
- While at the location, select every Nth passer by to ensure randomisation
7. Process and analyse the data
Unless you have been entering responses directly into a computer programme, processing the data involves preparing and translating the data for analysis. It involves taking the completed questionnaires and putting them into a format that can be summarised and interpreted.