Choosing the Right Research Methodology [Part 2]

telephone survey methodologies

Last week in this blog, we talked about the three main types of marketing research methodologies and focused in on the most popular forms of self-completed methodologies.  This week, we continue the discussion with interviewer-assisted and observation methodologies.

Telephone Surveys

Although declining in popularity, telephone interviewing continues to be an important data collection method for survey research. Telephone surveys can elicit a better response rate than other methodologies, simply because the interviewer can address some of the respondent’s objections and encourage their participation. Similarly, responses to open-end questions can be more thorough because live interviewers can probe the respondent’s answer, providing clarification and depth. Also, some target respondents, such as in B2B research, can only be reached by telephone.

Telephone interviewing has far fewer prohibitions on who may be contacted than online surveys and geographically representative samples are readily available, either through listed sources or through Random Digit Dialing (RDD).

So, why is telephone interviewing in decline?  The main reasons are that alternative methods are generally faster and less expensive.  Just as door-to-door interviewing gave way to phone surveys, so too have self-administered methods such as online and mobile methods replaced interviewer administered surveys such as telephone.

Other issues impacting the use of telephone surveys are the declining percentage of landline home telephones and the increasing incidence of people using their mobile phones as their only telephone.  Sampling from only landline phones could introduce a bias into your results (as consumers who still have landline phones tend to be older and less tech savvy than mobile-only respondents.) Also, because of the preponderance of caller-ID, even those people who do have landline phones are screening their calls and are reluctant to answer calls from unknown sources such as telephone interviewers.   Telephone interviewing is more expensive, because so many dialings have to be made to reach cooperative respondents. Compared to online surveying, variable costs for phone surveys are very high, so that large scale studies can be prohibitively expensive.

Including mobile phone numbers in your sample helps to reduce the bias, but introduces additional costs because regulations prohibit auto-dialing to mobile phones. That means that each mobile phone number must be dialed manually. (Note that the Marketing Research Association is lobbying for the marketing research industry to be exempt from the rule against autodialing mobile phones. So stay tuned for more on this topic.)

Another drawback to telephone surveys is the live interviewer. Interviewers are human beings, who can introduce an element of variability into the data collection. Professional marketing research firms train interviewers to read each question exactly as written, but whether due to boredom or fatigue or frustration, some interviewers will divert from the script, and possibly change the original meaning of the question.  What if the interviewer speaks accented English? Or if the respondent is slightly hard of hearing? Or if one or the other is having a bad day? There is also some evidence that respondents are less negative in their responses if they are speaking with a live interviewer than they are in a self-completed methodology, which can skew results.

Another consideration of using telephone surveys is that it does not allow the respondent to see what the interviewer is discussing.  So, if you are evaluating a product concept or a print ad, you can’t show the respondent a picture or written concept to evaluate.  Finally, some analytic techniques require long series of similar-sounding questions, which can be difficult for respondents to listen to and answer. Max Diff, Conjoint and some other techniques are generally only executed with online surveys.

In-Person Interviewing

Other interviewer-administered data collection methodologies include intercept or exit surveys, door-to-door interviewing, and central location tests. All of these other interviewer-assisted methodologies can be challenged by not being able to collect data from a representative sample of respondents.  For example, conducting a survey by intercepting shoppers at a mall will have a sample skewed to the demographics of the people in that geographic area.  Additionally, again due to the labor costs involved with live interviewers, costs of in-person interviewing can be very high.

Observation (Ethnography)

Another important interviewer assisted methodology is broadly known as “observation.” Using the observation technique, the interviewer does not intervene in the situation at all, but merely observes and records the participant’s behavior.  However, a bias may be introduced in this technique because the respondent, knowing he/she is being watched, may alter their “natural” behavior as a result.  This may be overcome somewhat through observing through videotape or some other mechanical device (such as recording key strokes on a computer).  (Observation is commonly used in retail shopper behavior studies, or in any situation where actual behavioral metrics are required (traffic studies, waiting behavior, television viewing behavior etc.) If done in person, cost of observation research can be prohibitive. Of course, if the observation is done without human intervention, the costs become more reasonable. Usually, the sample sizes will be relatively small with observational studies, given the amount of time needed to collect and analyze the behavioral data. This results in data that are more qualitative in nature and not projectable to a general population or market segment.

Making the Right Choice

The most important thing to understand in choosing a research methodology is that there is no single correct answer. The decision must be addressed individually for each project, and should be made based on a careful consideration of all available methodologies.  The key questions in the evaluation are:

  • How can you reach your respondents? If you have a list of respondents, what information do you have? If you only have landline telephone numbers, you may be forced to do a telephone survey. If you have email addresses, have the people on the list been given the opportunity to opt-out of receiving a survey solicitation?   If you are going to purchase a list for access to respondents, what information will that list have and will that force you to choose one methodology over another?  Do you need to obtain a sample from an online panel?  If so, how large and representative is that panel?
  • How long is your survey? In general, the shorter the survey the higher the response rate, and the better quality the resulting data. But some analytic methodologies require a more complex questionnaire. Most researchers agree telephone surveys should not exceed 15 minutes, online surveys should not exceed 10 minutes, and Mobile surveys should be 10 questions or less. Survey length is a key factor in methodology selection.
  • What kind of questions are you going to ask? Do you need to ask a lot of matrix questions with long lists of attributes? Are you using Max Diff or Conjoint or some other complex analytic technique? Do you need to show the respondent a lot of text, graphics, pictures, or video? You can’t display graphics or read extensive text on a telephone survey and repetitive or matrix questions are difficult on the telephone. Mobile surveys’ smaller screens do not handle graphics or long, wordy questions well. Do you plan to have a lot of open-end questions? Well-trained telephone interviewers usually produce richer, more complete open-end responses.
  • How much time do you have? Some research methodologies can be completed overnight and others can take week or months to complete. Be realistic in your expectations.
  • How much is your budget? Finally, different methodologies do come with different price tags. And while price may not be the most important factor, it is a reality all marketers must consider in their trade-offs.

The goal is to select the methodology that has the best chance of reaching the desired population, getting them to answer the survey, minimizing various types of biases, and achieving the research objectives and information goals. There is no perfect solution, just as you will never achieve 100% response rates. However, your choice of methodology can have huge implications for getting the information needed to answer your business questions. Be a student of research methodology so you can make the best choice for each project.

Leave a Reply

Your email address will not be published. Required fields are marked *