While no one in marketing research likes to talk about it, sample quality is a huge issue. Both in terms of sourcing sample and subsequently getting responses from qualified respondents, the industry as a whole is struggling. And while, especially for online sample, the number of companies managing and selling panel sample has increased, as an industry we have not yet developed any cohesive response or solutions to this problem.
The industry has pointed fingers at many reasons for declining sample quality, but no one reason stands out. Consumers are concerned about privacy, the proliferation of technology makes it difficult to positively identify respondents, the marketing research industry continues to rely on surveys that are too long, too invasive, and too tedious to maintain respondent engagement, among many other challenges.
The response for most marketing researchers is to vet their sample provider thoroughly. ESOMAR, the “membership organization representing the interests of the data, research and insights profession at an international level,” recently released their publication on this topic “28 Questions to Help Buyers of Online Sample.” While, as the name suggests, these are questions about acquiring online sample, many of them also apply to phone sample as well. We are not going to review all of them, but here are a few that are very worthy of consideration and implementation.
- All lists are not created equal. The source of information is key to ascertaining sample quality. ESOMAR recommends asking your sample provider to “describe and explain the types of online sample sources from which [they] get respondents.” Some sources might include existing databases, other actively managed panels, direct marketing lists, social network solicitations, and web intercept (also known as river sampling).
- Ensuring validity with multiple sample sources. Ask your sample provider to explain their process for ensuring validity of the sample if they draw from multiple diverse sources. Sample sources may contain the same potential respondents, causing respondent confusion and frustration, and ultimately reducing data quality. Determine if your sample provider goes beyond de-duping to truly validating the individuals.
- Avoid direct marketing. Recently, many sample providers have been using their samples to develop lists for direct marketing. This may cause lower response to marketing research projects if sample members are expecting to be sold something.
- A needle in a haystack. Some samples are harder to source than others. If you are looking for pet owners, you probably don’t have to worry. However, what if you want to talk to public transit riders in Gwinnett County, GA? It is a much more difficult (and hence more expensive) sample, and not all sample companies are up for it. And be sure to find out how your provider sources supplemental sample from other partners if needed.
- Using a Router. A survey router is a software system that allocates potential respondents to surveys for which they are likely to qualify. Perhaps they did not qualify for a previous survey they were invited to take, or perhaps they received a general invitation to “take a survey.” While there is no hard evidence that using a survey router is good or bad, be sure you understand how the surveys are being allocated. Evaluate the process against your specific project requirements for respondent qualification to make sure the router allocation is not introducing bias into your sample.
- Who are these people? One of the benefits of online samples is the extensive profiling data that companies hold to assist in sample selection and to ensure the resulting samples are representative of the desired population. Be sure to ask what profiling data is available – including on respondents from different sources and partners – and how your sample partner is keeping their profiling data up-to-the-minute. Depending on your project requirements, you may need to evaluate how these profile questions are asked to ensure their value to you.
Given that sample is one of the key fundamentals for high-quality research, you owe it to yourself to be hyper-informed. Many researchers order sample as if it is a “black box” or a commodity, but the specific policies and procedures of the sample vendor can have a huge impact on your project. This is simply too important to leave unexamined! Dig in and learn – and ESOMAR’s publication might be a good start.
We will publish more tips on purchasing online sample in our next blog. Don’t miss it!
Concerned about sample quality? We are too. Contact Us, and we can help!