Let’s face it. Surveys are getting a bad name.
Despite having served marketers well for decades as a valuable tool, the quality of many survey research projects has decreased, and the entire field has come under attack by marketing scientists and academics. Declining response rates caused by concerns for privacy and general lack of public support for marketing research, changing cultural, demographic, and technological trends, the proliferation of panels that are less than representative, badly designed and analyzed DIY surveys, election polls that fail to predict – is it any wonder that businesses are challenging the necessity and validity of survey results? Add in the relentless time pressures faced by businesses, the belief that survey research costs too much (if done properly), and the seductive allure of Big Data, and you have a perfect storm of forces accelerating doubts about the value and validity of survey research.
One of the reasons that surveys were so popular and well-used in Marketing was that they were one of the only sources of data available. And they were a single source for a lot of information. As Peter V. Miller writes in AAPOR’s Public Opinion Quarterly, “In addition to measuring key phenomena of interest (e.g., Web use, voting, employment, health status), surveys also offer the chance to capture data that may help further understand those phenomena. This covariate information—demographic and lifestyle characteristics, attitudes, access to healthcare, and so forth—allows the survey analyst to see how different factors relate to the key variables.”
Despite their value, surveys would probably not have been the first-choice data source if other sources had been available easily and cost-effectively. Now, survey research’s weaknesses can be addressed by blending survey data with data from alternative sources. For example, the decennial Census has been traditionally a survey of every household in America. Originally a door-to-door sample, the most recent 2010 Census was a combination of self-reported and in person non-response follow-up. As these methods have become very costly, plans for the 2020 Census include using alternative data sources (e.g., tax filings, health records, etc.) to count some households without costly in-person follow-up. And the JASON 2016 advisory group of the Department of Defense recommends the 2030 Census be conducted mostly through administrative data, using surveys only to fill in data that are not available through other sources, a practice that is widely used in other countries but not yet in the U.S.
However, while these data sources are available and cost-effective, for them to become more widely useful to businesses, there are several obstacles to overcome. Here are the key obstacles we must solve, according to Miller:
- Same data labels, different measures. When blending data sources, care must be taken to ensure that data labels are equally descriptive of the metric in question. For example, asking someone if they are disabled in a survey may get a different result from having a data record identifying a person as disabled. The strict legal construct of disability that meets regulatory standards may differ from someone’s self-perception and self-reporting.
- Access versus Privacy. Even when trying to access confidential or sensitive information within their own companies, researchers may be stymied by the obstacles that must be overcome to get access to the data. Is allowing access justified by the use? Will sensitive information continue to be protected at the same level? Even the researchers bona fides come into question in this process.
- Aggregating data from different geo-boundaries. Voting precincts and Census blocks may differ in some states because the Census block is defined in collaboration by state officials and federal demographers in advance of each Census. However, that does not happen in all states, in which case geographic boundaries may not be reconcilable for data aggregation.
Interestingly, these obstacles are not that different from recent challenges to survey research. While these challenges are not insurmountable, they are extremely challenging. These challenges will eventually be resolved, through changing public policy, the creation of identity-free microdata sets, and even technological and analytical advances. And surveys will still be needed to fill in the information gaps that remain. So, for the meantime, while we work on solving the data access challenges, we need to continue to improve survey research methods as well.