How to Protect the Integrity of Survey Research

Science requires data, and survey research is one important means of gathering it. Surveys provide a scientific way of acquiring information that is used to inform policy decisions, guide political campaigns, clarify the needs of stakeholders, enhance customer service, help society understand itself, and improve the quality of life in the United States.

In recent years, concerns have been raised about growing rates of refusal to participate in surveys, as well as about inaccurate forecasts in high-profile elections, polls with contradictory findings, the declining trust in government and media institutions that fund such research, and skepticism fueled by political polarization. “Although polling is not irredeemably broken,” the authors of a new article write, “changes in technology and society create challenges that, if not addressed well, can threaten the quality of election polls and other important surveys on topics such as the economy.”

In this article, published in the journal PNAS Nexus, 20 experts from diverse fields — including academia, science, government, nonprofits, and the private sector — offer a dozen recommendations to improve the accuracy and trustworthiness of surveys.

Transparency, clarity, correcting the record

The authors’ recommendations aim to better align the practices of survey research with three scientific norms: transparency, clarity, and correcting the record. Among the recommendations are that surveyors and survey researchers should:

• Be transparent about their methods, including sampling design and modeling and weighting assumptions.
• Disclose the sources of respondent recruitment.
• Disclose the ways in which exposure to other surveys may affect the responses of members of panels.
• Disclose the known or expected consequences of attrition on panel surveys.
• Be more precise in the use of terms such as “representative sample” and spell out what that means.

“If survey panelists’ responses have been potentially biased by responses to earlier surveys or questionnaires, researchers and readers need to know that,” said co-author Kathleen Hall Jamieson, director of the Annenberg Public Policy Center of the University of Pennsylvania and a professor of communication at the university’s Annenberg School for Communication.

The authors also recommend the creation of a publicly available, professionally curated archive of identified technical problems in polling tools and their remedies. And they recommend that professional organizations and universities develop and disseminate a guide to survey research for use in high school courses as well as a brief “users guide” designed for laypeople and journalists who do not have training in statistics or survey research.

“Around the world, survey research produces vital information,” said co-author Arthur Lupia, a distinguished professor at the University of Michigan. “These new recommendations have the potential to increase the quality of future surveys for generations to come.”

The recommendations emerged from a virtual two-day retreat in November 2021 convened by Marcia McNutt, president of the National Academy of Sciences, and co-hosted by The Annenberg Foundation Trust at Sunnylands and the Annenberg Public Policy Center. The sessions were coordinated by Lupia and Jamieson, who also is the Sunnylands program director. Among the participants were the authors of this paper, who include past and current editors of major academic journals, past presidents of the American Political Science Association and the American Association for Public Opinion Research (AAPOR), a past director of the U.S. Census Bureau, scholars who have led some of the nation’s largest university-based election surveys, and individuals responsible for the creation and maintenance of major government and nonprofit survey datasets.

Read the full article, entitled “Protecting the integrity of survey research” in PNAS Nexus.