Full file at

CHAPTER 2

Techniques, Tools, and Tactics

LEARNING OBJECTIVES

After reading and studying Chapter 2, students should be able to:

  1. Understand the requirements, limitations, and methods of psychological research.
  2. Explain the experimental method, how to design an experiment, and how to select participants in research.
  3. Explain the use of naturalistic observation, polls, and surveys to collect data.
  4. Describe how to construct a representative sample.
  5. Understand the basic concepts in descriptive statistics, inferential statistics, and meta-analysis.

KEY TERMS

  1. Control group – in an experiment, this is the group of research participants that is not exposed to the independent variable.
  2. Correlation – the relationship between two variables. The strength and direction of the relationship is expressed by the correlation coefficient.
  3. Dependent variable – in an experiment, this is the resulting behavior of the subjects, which depends on the manipulation of the independent variable.
  4. Descriptive statistics – ways of describing or representing research data in a concise, meaningful manner.
  5. Experimental group – in an experiment, this is the group of research participants that is exposed to the independent variable.
  6. Experimental method – the scientific way to determine the effect or influence of a variable on the subjects’ performance or behavior.
  7. Fixed-alternative survey questions – survey questions to which respondents limit their answers to the choices or alternatives presented. They are similar to multiple-choice questions on college exams.
  8. Independent variable – in an experiment, this is the stimulus variable that is manipulated to determine its effect on the subjects’ behavior.
  9. Inferential statistics – methods for analyzing research data that express relationships in terms of probabilities.
  10. Mean – the arithmetic average; a way of describing the central tendency of a distribution of data.
  11. Median – the score at the midpoint of a statistical distribution; half the scores fall below the median, and half above.
  12. Meta-analysis – the large scale reanalysis of the results of previous research studies.
  13. Mode – the most frequently obtained score in a distribution of data.
  14. Naturalistic observation – the scientific observation of behavior in its natural setting, without any experimental manipulation of the independent variable.
  15. Normal distribution – a bell-shaped distribution of data in which most scores fall near the center and few fall at the extreme low and high ends.
  16. Open-end survey questions – survey questions to which respondents state their views in their own words. They are similar to essay questions on college exams.
  17. Probability – the idea that the differences between the means of experimental and control groups could have occurred by chance.
  18. Probability sampling – a method for constructing a representative sample of a population for surveys or polls. Each person in the population has a known probability or chance of being included in the sample.
  19. Quota sampling – a method for constructing a representative sample of a population for surveys or polls. Because the sample must reflect the proportions of the larger population, quotas are established for various categories such as age, gender, and ethnic origin.
  20. Skewed distribution – an asymmetrical distribution of data with most scores at either the high or the low end.
  21. Standard deviation – a measure of the variability of a distribution; the standard deviation is a precise distance along the distribution’s baseline.
  22. Statistical significance – the level of confidence we can have in the results of an experiment. Significance is based on the calculation of probability values.
  23. Survey research method – interviews, behavioral observations, and questionnaires designed to sample what people say about their feelings or opinions, or how they say they will behave in a given situation.

REVIEW QUESTIONS

  1. Why would knowledge of psychological research methods be of value to you in your working career?

a.You will probably have to deal with the findings of I-O psychologists

b.To use their advice you must understand how they studied the problems and arrived at their conclusions

c.The goal is to acquaint you with the requirements, limitations, and methods of the scientific approach.

  1. What are three basic requirements of psychological research? What are the limitations of psychological research?
  2. Requirements

i.objective observation

ii.control

iii.duplication and verification

  1. Limitations

i.not all behavior can be studied

ii.observing behavior can change it

iii.the Hawthorne effect

iv.artificial settings

v.college students as subjects

  1. Distinguish between (a) dependent and independent variables, (b) random group design and matched group design, and (c) experimental groups and control groups.
  2. Dependent and independent variables

i.Independent variable: the stimulus variable that is manipulated to determine its effect on the subject’s behavior

ii.Dependent variable: the resulting behavior of the subjects, which depends on the manipulation of the independent variable

  1. Random group and matched group design

i.Random group design: subjects assigned at random to experimental and control groups

ii.Matched group design: subjects in one group are matched with subjects in the other group on characteristics that could affect their performance

- desirable
- costly and time consuming
  1. Experimental groups and control groups

i.Experimental group: exposed to the independent variable

ii.Control group: not exposed to the independent variable, but similar in every other way

  1. How would you design an experiment to determine why experienced computer operators who had worked for the company five years suddenly begin making data-entry mistakes when their department was moved to a new workplace?

a.Example: Designing an Experiment regarding location and mistakes

i.Independent variable

1.Location

ii.Dependent variable

1.Mistakes

iii.Element of control

1.Experimental group

a.Exposed to the independent variable (new location)

iv.Control group

1.Not exposed to the independent variable (i.e., remain in old location), but similar in every other way
  1. Describe advantages and limitations of the naturalistic observation method.
  2. Naturalistic observation – the scientific observation of behavior in its natural setting without any experimental manipulation of the independent variable
  3. Advantages
  4. Behaviors being observed are typical of what occurs daily
  5. Findings can be more readily generalized and applied
  6. Limitations
  7. Without control of the independent variable, it’s difficult to conclude with assurance what caused any changes
  8. Observation cannot be repeated
  9. What are some of the uses of surveys and public opinion polls? Discuss some of the problems that can limit their usefulness.
  10. Uses
  11. Ascertain what factors contribute to job satisfaction and morale
  12. Gather information on work-related issues
  13. Assess reactions to changes in the workplace
  14. Allow for employee participation in policy making
  15. Uncover consumer preferences
  16. Assess voter reaction to candidates and issues
  17. Problems with surveys

i.Difficult to measure precisely our personal opinions and attitudes

ii.Some people deliberately lie

iii.Socially or politically correct response

iv.It’s difficult to find people to respond

v.Phrasing of the question often influences the response

  1. Describe four methods used to collect survey data.

a.Personal interviews – higher response rates than other methods; interviewers can clarify questions; some people are uncomfortable with personal interviews, and not all people are equally accessible

b.Questionnaires

i.Most frequently used - and abused - method

ii.Paper or computer-assisted

  1. Web-based – there are age and cultural differences in preference; often employees are questioned too frequently; there is concern over “ballot stuffing”
  2. Telephone surveys – telephone response rates have plummeted over the last decade
  1. If your job was to conduct a market survey of BMW car owners to find out what changes they wanted to see in new models, what survey research method would you use? Why?
  2. Use a combination of means to collect data: focus groups to personal interviews of current owners; email surveys; telephonic; and web-based collection. Each method has its own strengths and blind spots. Offering incentives will increase the useful response rate. Focus groups allow for more in-depth and follow-up on questions. Random selection of customers is a must. Control for differences in type of car and geographic location.

  1. What are push polls? How can they influence the outcome of elections?
  2. Push polls use false or distorted information to see if certain information can push voters away from one candidate in favor of the other (who is paying for the poll).
  3. Differentiate between open-end and fixed-alternative survey questions. Discuss the differences between probability sampling and quota sampling.
  4. Open end

i.Respondents state their views in their own words

b.Fixed-alternative

i.Respondents must limit their answers to the choices presented

ii.Simplifies the process
iii.May not completely reflect respondents’ range of opinion
  1. Describe advantages and limitations of conducting psychological research in virtual laboratories.
  2. Advantages
  3. Run faster and less-costly studies
  4. Draw on larger and more diverse pool of subjects
  5. Data can be saved automatically in electronic form
  6. Recruiting research subjects with a broad range of personal characteristics
  7. Limitations
  8. Impossible to assess the honesty and accuracy of responses
  9. Computer literate people tend to have higher levels of education, different from the population at large
  10. Respondents are self-selected
  11. Response rate is lower and drop-out rate is higher
  12. “Ballot stuffing” is a possibility
  13. What is the difference between descriptive statistics and inferential statistics?

a.Descriptive statistics

i.Ways of describing research data in a concise, meaningful manner; usually focused on one group at a time

  1. Mean, median, mode
  2. Standard deviation
  3. Correlation

b.Inferential statistics

i.Used to compare performance levels of the experimental group and the control group

  1. Determine the level of statistical significance of the difference between the means of the two groups
  2. Calculate a probability value for the difference between the means
  1. In your opinion, would the world be a better place without statistics? Defend your answer, showing appropriate data.
  2. Each student will have their own answer. However, without statistics we would be unable to compare and contrast, evaluate program effectiveness, performance, or provide meaningful feedback.
  3. What can the standard deviation reveal about your performance on an IQ test?
  4. We can determine the percentage of scores that fall above or below a specific score. The average IQ score is 100 and the SD is 15. For example, 99.5 percent have scores less than 145 (+3 SD); 97.5 percent below 130 (+2 SD); 84 percent below 115 (+1 SD).
  5. We can compare the relative performance of individuals
  6. We can tell where an individual score falls, relative to the group as a whole
  7. What does a correlation of +.85 tell us about the relationship between your IQ score and your boss’s rating of your job performance?
  8. It suggests a positive relationship in the direction of the relationship between the two scores (i.e., as one score increases, so does the other)
  9. It suggests a strong relationship between the two scores
  10. It does not imply causality

PRACTICAL EXERCISES

  1. Many students are confused by the terminology used in the experimental method. Divide the class into groups of 3-5 students each. Have the groups design a simple experiment to determine the affect that sleep deprivation will have on examination performance. Each group should select a secretary who will present their results at the end of 10-15 minutes (using butcher paper, white board, or overhead transparency). Their presentation should specify: dependent variable, independent variable, experimental group(s), control group, and method of ensuring a representative sample.
  1. Meta-analysis is defined as the large-scale reanalysis of the results of previous studies. Outside of class you can ask students to review a journal, e.g., Journal of Applied Psychology, for examples. In class you could demonstrate using the following examples.

Suppose you are the human resource manager for a manufacturing plant and are interested in incorporating a new test on mental abilities as part of your overall test battery for hiring new personnel. Your current test has a 0.25 correlation with performance. Before spending time and money implementing the new test, you conduct a review of the literature and find several studies correlating the new test with employee performance. Based on the meta-analysis done below, what can you conclude?

STUDY

/

N

/ r /

N*r

Smith (1997) / 125 / .22 / 27.5
Jones (2000) / 250 / .33 / 82.5
Johnson (1999) / 33 / .35 / 11.55
Meriwether (2001) / 99 / .18 / 17.82
Total / 507 / 139.37
Mean Correlation / .275

In this correlation example, meta-analysis is nothing more than the average effect size weighted by sample size. You may either work the above example in class or present it (or a similar mock-up) to students, leaving the total and mean correlation figures blank, and ask the students to conduct the meta-analysis.

Other meta-analysis studies may compare the results of an experimental group and a control group. In this instance, the effect size d = (Me-Mc)/SD, where Me = mean of the experimental group, Mc = mean of the control group, and SD = standard deviation. This data is usually contained in the published study.

Using this approach, for example, suppose a plant manager wanted to study the effect that a new training technique might have on employee production rates. Her literature search found several studies on the subject (shown below). Based on meta-analysis, what do you conclude?

Type of Training
STUDY / New Technique (Me) / Control (Mc) / SD / N / D / N*d
Schultz (1998) / 33 / 42 / 5 / 42 / -1.8 / -75.6
Meriwether (1995) / 777 / 763 / 14 / 67 / 1 / 67
Smith (1999) / 500 / 444 / 20 / 100 / 2.8 / 280
Jones (2001) / 67 / 75 / 15 / 115 / -.53 / -61.3
Total / 324 / 210.1
Mean effect size / .64

In the preceding example, the effect size is considered moderate. According to J. Cohen (Statistical power analysis for the behavioral sciences. New York, Academic Press, 1977), an effect size less than .4 is considered small, between .4 and .8 moderate, and greater than .8 is large.

TEST ITEMS

TRUE-FALSE ITEMS

  1. Some problems are too complex to investigate by psychological research methods. (T, p. 31)
  1. The act of observing some behaviors may interfere with them. (T, p. 31)
  1. The novelty of change may affect behavior more than the actual change itself. (T, p. 32)
  1. The generalizability of findings in laboratory settings is unlimited. (F, p. 32)
  1. In the experimental method, the variable being investigated is the dependent variable. (F, p. 32)
  1. In the experimental method, the behavior that results is the dependent variable. (T, p. 33)
  1. Assigning subjects at random to the experimental and control groups is brought about by matched group design. (F, p. 34)
  1. Survey research involves observing behavior in its every day setting. (F, p. 38)
  1. Survey questions must be objective. (F, p. 39)
  1. Fixed-alternative survey questions limit respondents to a few choices. (T, p. 44)
  1. The central tendency of a distribution is measured by the standard deviation. (F, p. 51)
  1. In a normal curve, 68 percent of the population falls within plus or minus one standard deviation from the mean. (T, p. 51)
  1. A negative correlation coefficient depicts an inverse relationship between two variables. (T, p. 52)
  1. A technique for averaging the results of a large number of studies is meta-analysis. (T, p. 53)

LISTING/SHORT ANSWER QUESTIONS

  1. What are three requirements of psychological research? (pp. 30-31)

a.Objective observation

b.Control

c.Duplication and verification

  1. What are limitations of psychological research? (pp. 31-32)

a.Not all behavior can be studied

b.Observing behavior can change it

c.The Hawthorne effect

d.Artificial settings

e.College students as subjects

  1. List four methods of psychological research. (pp. 32-46)

a.Experiments

b.Naturalistic observations

c.Polls and surveys

d.Web-based research

  1. Describe the experimental method. (pp. 32-33)

a.The scientific way to determine the effect or influence of a variable on the subjects’ performance or behavior.

i.Independent variable

1.The stimulus variable that is manipulated to determine its effect on the subjects behavior

ii.Dependent variable

1.The resulting behavior of the subjects, which depends on the manipulation of the independent variable

iii.Element of control

1.Experimental group

a.Exposed to the independent variable

2.Control group

a.Not exposed to the independent variable, but similar in every other way
  1. Compare and contrast random group design and matched group design. (p. 34)

a.Control group and experimental group must be as similar as possible

i.Random group design

1.Subjects assigned at random to experimental and control groups

ii.Matched group design

1.Subjects in one group are matched with subjects in the other group on characteristics that could affect their performance

2.Desirable
3.Costly and time consuming
  1. Define naturalistic observation and identify advantages and disadvantages of its use. (pp. 36-37)

a.Observation of behavior in natural setting without experimental manipulation of independent variable

i.Avoids artificiality

ii.Difficult to conclude what brought about change

iii.The observation cannot be repeated

  1. Discuss the problems with surveys. (p. 39)

a.Difficult to measure precisely our personal opinions and attitudes

b.Some people deliberately lie

c.Socially or politically correct response

d.It’s difficult to find people to respond

e.Phrasing of the question often influences the response

  1. Identify three basic ways to collect survey data. (pp. 39-42)

a.Personal interviews

b.Questionnaires

i.Most frequently used - and abused - method

ii.Paper or computer-assisted

c.Telephone surveys

  1. Contrast open-end survey questions with fixed-alternative survey questions. (p. 44)

a.Open-end

i.Respondents state their views in their own words

b.Fixed-alternative

i.Respondents must limit their answers to the choices presented

ii.Simplifies the process

iii.May not completely reflect respondents range of opinion

  1. Specify the difference between probability sampling and quota sampling. (p. 45)

a.Probability sampling

i.Each person in the population has a known probability of being included in the sample

b.Quota sampling

i.Because the sample must reflect the proportions of the larger population, quotas are established for categories such as age, gender, and ethnic origin

  1. List and define the three basic measures of central tendency. (p. 48)

a.Mean (arithmetic average)

b.Median (score at the midpoint of a statistical distribution; best statistic to use with skewed distributions)