More than a decade ago, several bright minds in the marketing research world did some research-on-research and decided that 20 minutes was the right length for a survey. Balancing response rates, data quality, and respondent engagement, 20 minutes was the winner. So that’s what we did. We based all our evaluation criteria on achieving a 20-minute “LOI” (length of interview). So, not surprisingly, the average length of an online or phone survey currently is 20 minutes across the marketing research industry.
But is that still the right standard? The average mobile survey takes 8.3 minutes. Are people really willing to spend more than twice as long completing a survey with a live interviewer or on their computer as they are using an app on their phone? Somehow, when you think about all the changes in society and in technology over the past decade, you must question: should we still be aiming for that 20-minute LOI? Or is 8.3 the better answer? And if not those options, what is the optimal survey length?
As you might have guessed, the optimal survey length depends on many, many factors. Here are a few:
- What’s the purpose of your research? Doing a “deep dive” into a topic of consumer behavior may take longer than determining which of two ads performs better on a set of metrics.
- Who is your respondent? Busy professionals may be reluctant to spend 20 minutes completing a survey, at least without adequate compensation. On the other hand, a consumer surfing the net while watching television might very well take 20 minutes for a survey. A panel member who is being paid to take surveys and who understands the survey process will evaluate the transaction and determine if it is justified.
- What’s the topic? A short, interesting topic will generate a higher response rate than a highly technical but somewhat dry topic. Simple questions that can be answered quickly will also perform better on response rates. A topic that clearly delivers potential upside to the respondent will also be more likely to get completed.
- What’s your methodology? Convenience and engagement are paramount. Telephone surveys are very intrusive – they never seem to come at a good time! Mobile and online surveys can be accessed when convenient for the respondent, but may also be delayed indefinitely. Gamified surveys and prediction models can entertain the respondent during data collection.
Balancing all these factors to find the optimal survey length is challenging. And the rule of thumb that most researchers agree on today is simple: Keep it as short as possible. You want to collect the minimal information you need to make decisions. Here are some tips for shortening surveys:
- Review every question. If it hasn’t changed, won’t be used in analysis, or can be answered by means other than the survey, eliminate that question.
- Ask sample providers to supply geographic, demographic, and other information so you don’t have to re-ask it in the questionnaire.
- Break the survey up into several chunks to be completed by different sets of respondents (or the same respondents at different times).
- Use gamification and variety to reduce cognitive burden and lessen fatigue.
- Pretest your survey to make sure the questions are properly worded, make sense, follow a logical order and that responses are appropriate and complete. Your pretest should include asking respondents to take the survey, but also asking them about their experience in taking it.
- Minimize the number of open-end questions you ask.
- Stick to the research objectives and avoid asking any additional questions “as long as we have the respondents in the survey.” A survey is not a Christmas tree – more is not better.
No matter how long you decide your survey should be, there is one thing you can count on: your respondents will perceive it to be much longer than it is! And when respondents start to feel beleaguered, data quality suffers. Check your data quality thoroughly after it is collected, and even oversample to make sure you end up with enough high-quality completed surveys to meet your analytical goals.