Chapter 11

In the six preceding chapters, we have used the results of survey research to

describe the origins of opinions on political issues, to discuss the public’s orientation

toward the political system, and to examine public opinion on social-welfare,

racial, cultural, and foreign policy issues in an attempt to describe the American

mind. This chapter explores some of the challenges that face survey researchers and

identifies a number of factors that should be considered in deciding how much

confidence to place in the findings reported in a public opinion survey.

As part of its special 50th issue, the editors of The Public Perspective (1998b:

86–94) asked nine senior survey practitioners to write about ‘‘the greatest challenges,

or problems, now confronting public opinion research in the U.S.’’ Although

each of these researchers cited a different aspect of the survey profession, the theme

running throughout these articles addressed the reliability and trustworthiness of

polls. While the specific challenges identified included the quality of data collection,

response rates, interpreting results appropriately, and reporting them

better, they all touched on the need to produce valid data. As Newport (1998: 87)

noted, ‘‘the reliance on survey results to guide the ship of state necessitates that

polls be trustworthy and reliable when they are first published.’’ In order for

public opinion data to be trustworthy, they must be reliable, producing consistent

results, and valid, measuring what they intend to measure.

Even though it is somewhat ironic to attempt to measure the public’s opinion

about polls using data from a public opinion poll, previous research has shown

that while Americans have some reservations about poll results, they generally

view surveys in a positive light (Goyder, 1986; Kohut, 1986; Roper, 1986; Dran

and Hildreth, 1995).

Prior investigations of this topic have shown that the public generally believes

that polls interview typical, representative people; are honest and accurate; work

for the interests of the general public; and are an enjoyable and satisfactory

experience (Roper, 1986: 10–13). Later research, such as that of Dran and Hildreth

(1995; 1997), found a more mixed view of public opinion on the polls, with less

than half of the public expressing confidence in their accuracy. A 1996 survey by

the Gallup Organization found an increase in the number of people who pay

attention to poll results. It also found that two-thirds of the public has faith in the

accuracy of surveys. In addition, almost three-fourths of their respondents said

that the country would be better off if the nation’s leaders followed public

opinion polls more closely, 87% felt polls are a ‘‘good thing’’ for the country, and

68% believed that polls work for the best interests of the general public (Gallup

and Moore, 1996: 51–52). By 1999, however, only 38% of respondents said they

had a good deal or fair amount of trust in what they saw or heard in public opinion

polls, 52% felt that the country would be better off if less attention were paid to

polls, and 54% believed that political officeholders and public officials paid too

much attention to polls (Gallup Report, 1999).

The Council for Marketing and Opinion Research (CMOR) periodically

conducts studies that examine the public’s image of surveys. These studies have

also found a mix of positives and negatives in citizen attitudes. On the positive

side, CMOR’s studies have shown the public to believe that the survey research

industry serves a useful purpose, that responding to surveys is in their best interest,

and that polls and surveys are useful in providing government officials with an

understanding of how the public feels about important issues. On the negative

side, the public believes that there are too many polls on subjects of little value, is

skeptical of results reported in surveys, and does not understand how the small

number of people interviewed in the typical survey can represent the entire

population. The most recent CMOR study, conducted in 2003, found that positive

perceptions of surveys were declining and negative perceptions were increasing.

Although the public still feels that participating in a poll is a pleasant experience

and generally finds the subject matter of polls interesting, there has been a marked

increase in public concern over the issue of privacy. A higher percentage of

the public now feels that polls are an invasion of privacy and a large majority does

not believe that organizations that conduct polls and surveys can be trusted to

protect their privacy rights. While survey researchers diligently protect the

confidentiality of respondents, the industry rules related to privacy issues are not

being communicated clearly or are not being understood by the public (Council

for Marketing and Opinion Research: 2003).

In 2001, the Henry J. Kaiser Family Foundation commissioned a large-scale

study ‘‘to explore Americans’ opinions about public opinion polling, to see whether

polls are regarded as an effective instrument for expressing the people’s will

to leaders in government’’ (Brodie, Parmalee, Brackett, and Altman, 2001). The

results of this study were also a mix of good and bad news for survey researchers.

Americans overwhelmingly believed that the will of the majority should influence

the decisions of government officials and felt that public opinion polls are a good

way for public policy makers to learn what people think. Moreover, three-fourths

of the public thought that decision makers should pay a great deal or fair amount

of attention to public opinion polls in considering important issues. On the

negative side, ‘‘majorities of the public think that polls are inaccurate, have inherent

limitations that prevent them from communicating the public’s views, and

are subject to manipulation. While half believe that they are based on sound scientific

practices, half disagree or are not sure about the polling process’’ (Brodie, Parmalee,

Brackett, and Altman, 2001: 13). These results are now five years old, and factors

such as the increased difficulty in gaining respondent cooperation in all surveys

and results such as those from the CMOR study would lead one to suspect that

this mix of good and bad news would be tilted more in the negative direction if

this study were repeated today.

There is some skepticism in the American mind about how survey data are

collected and how such information is used and reported. In recent years, increasing

concern over the confidentiality of information provided has contributed

to a decline in the public’s view of surveys. Despite the general confidence that the

public has in polls as a mechanism for expressing their views, a number of factors

can lead to errors in the way survey data are collected, interpreted, or reported

that might cause the public to lose even more faith in the polling process. In the

remainder of the chapter we examine several of these elements, including pseudopolls,

technological developments, and respondent factors.