ASM BSP research residency, July 16-19, 2008

Alix Darden – Surveys,

Surveys: Guidelines for Survey Development

1. Clearly define the objective of the survey. Keep your survey brief. This can be done by distinguishing what is "nice to know" from what is "need to know" and sticking with the "need to know". Make sure the questions are giving information that meets the objective of the survey.

2. Use simple language and common concepts:

  • Common words
  • Short questions
  • Avoid double-barreled questions (two questions in one), implicit negatives, overlong lists, and dangling alternatives.

3. Keep tasks manageable:

  • Ask personal fact questions. Respondents can answer questions of personal fact easier than questions of opinions and attitudes.
  • Establish clear definitions for terms of shared meaning (e.g., "family").
  • Avoid or narrow any recall of the past.
  • Avoid hypothetical questions (you get a hypothetical answer).

4. Question structure:

  • Use simple questions rather than general ones.
  • Use close-ended and open-ended items for appropriate inquiries.
  • Offer a "no opinion" option.
  • Use forced-choice response options rather than agree/disagree response options.
  • Watch question order, particularly for sensitive material.
  • Watch wording choices.
  • Use open-ended follow-ups to close-ended questions.
  • Ask multiple questions on an important topic.
  • Use skip patterns where appropriate.

5. Use the tools at hand:

  • Use professional experts to develop, administer, and analyze the survey.
  • Interview cultural insiders to understand diversity issues.
  • Borrow questions from others or use entire surveys that are already published.

6. Run a pilot test on a sub-sample (25 to 75 persons) to make sure respondents:

  • Understand the meaning of the questions.
  • Can respond to the questions (the tasks aren't too difficult).
  • Maintain interest and attention.

The pilot can be done with the interviewer in the room with the respondent to discuss the respondent's reaction to the survey.

If this is not done, within the pilot test, ask for feedback on the survey itself regarding:

  • Flow and naturalness of the sections
  • Order of the questions
  • Skip patterns
  • Timing
  • Respondent interest and attention
  • Respondent well-being (regarding sensitive nature of any of the items)

Adapted from: Converse, J. M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire.Newbury Park, CA: Sage Publications.

On-line tools

  • Can Google “student learning surveys”
  • Many books – search “survey research” at Amazon, etc.

SALG –Student assessment of learning gains

SENCER SALG – Science Education for New Civic Engagement and Responsibilities

Field-tested Learning Assessment Guide

VARK-Learning styles inventory

From: Central Michigan Assessment Toolkit – includes a section on surveys

CLASS- Colorado Learning Attitudes about Science Survey

Books – search your favorite bookseller for “survey research” and you will find a lot out there. The following is a recent, short, to-the-point book that gives an overview of the issues associated with surveys.

Fink, Arlene. 2006. How to conduct surveys. A Step-by-Step Guide. 3rd ed. Sage Publications, Thousand Oaks, CA. Fourth edition to be published 9/08.

SALG - The Student Assessment of Learning Gains Instrument is designed for instructors from all disciplines who wish to learn more about how students evaluate various course elements in terms of how much they have gained from them.

Feedback from the instrument can guide instructors in modifying their courses to enhance student learning. It may be used

at any point during a course (for formative feedback) as well as at the end it can be connected to course learning outcomes.

There are two surveys available: Baseline – designed to be used at the beginning of the semester.

SALG – designed to be used mid-end of the semester.

CLASS – Colorado Learning Attitudes about Science Survey – Carl Weiman along with several collaborators havedeveloped a survey instrument, the Colorado Learning Attitudes about Science Survey (CLASS) and have validated itwith extensive interviews and statistical tests in physics courses. They have observed a number of interesting features,including that most physics courses shift students’ attitudes in an unfavorable direction (away from “expert-like” andtoward “novice-like”). We see correlations between certain aspects of the students’ attitudes and certain aspects oflearning, as well correlations with their proclivities to drop a class or enroll in a follow up class. They have identifiedsome teaching practices that are clearly beneficial and some that are clearly detrimental to student’s attitudes aboutphysics.” The tool is currently available for physics and Chemistry and will soon be available for Biology. I have been told you can get the Biology one if you contact them.

1