“I need a representative sample.” Or do you?
As a marketing research vendor, we often get requests for a “nationally representative sample.” That has somehow become a buzzword for research quality. Many polls and surveys reported in the media purport to have “nationally representative samples,” thereby implying that they are sound. (Of course, bias could have been introduced into these surveys in question wording, or other ways, but we’ll leave that for another blog topic!)
By definition, a representative sample is a group of participants selected from a larger population that closely matches the characteristics of the population as a whole. In other words, the sample is a fairly accurate reflection of the population from which the sample is drawn. Nationally representative samples have characteristics that closely match the nation of interest.
Typically, to get this representativeness, you need a large sample size so you can reliably weight or balance your resulting data back into proportion with the population, without introducing additional bias. And we all know that the larger the sample size, the higher the cost. You must be careful what you ask for!
So, start with this: Who do you need to sample? If your research topic is found in the general population, a nationally representative sample might be appropriate. But if your research topic is only familiar to a subset of the population (e.g., Moms with children under 5, orthopedic surgeons, people who use your product category, etc.) you might waste a lot of money trying to get a representative sample. So your first step is to determine who you need to talk to, and then to figure out how best to talk to those people.
Caveat: A lot of marketing research is conducted with companies’ customer lists. And that is fine – except when it isn’t. If you want to find out why people don’t buy your products, or what they think of competitive brands, you might not be able to get this information from your customers. Lost customers and non-customers are also important sources of marketing information.
So the next question is sampling frame – or in simpler terms, the source for your sample. How representative of your desired population is your sampling frame? Let’s start with telephone numbers. About half of all US households have a landline telephone, and those households skew significantly older. Cell phones, on the other hand, while increasingly adopted as the sole phone for much of the population, will give you a sample that skews younger than the population. Online surveys have similar dangers. If you use a panel sample, does the simple fact that your sample is panel members mean that they may be different from your population in some way? One solution is multi-modal studies that draw participants from several sampling sources, but do the different methodologies introduce a bias?
There is no perfect sample. Indeed, there never has been. Perhaps more importantly, then, is to question: How good is good enough? Each project must be designed by balancing the optimal sampling frame and methodology against the budget and time available to complete the project. There are different standards for scientific research, for tracking studies, or for new product development research. As researchers and research users, you must accept that it will never be perfect. You must understand how your sample is deficient and protect your resulting data from any bias that might be introduced. This is achievable and should be your sampling goal.
There is no doubt that sampling is a critically important factor in getting sound research results for decision making. It is also an area that causes a lot of confusion among clients (and to be honest, among a lot of researchers, as well!) And, as one of the largest cost elements of marketing research, finding the right sample for the project is key to effective design and management. By establishing sound research objectives that clearly set out what information is needed, sample decisions should become a little easier.