Designing Questionnaires Respondents Won’t Hate [Part 1]

questionnaire designSurvey response rates have plummeted in recent years, for both consumer and business surveys. There are many reasons for this:

  • Concerns about personal privacy and information security
  • Time limitations (the ubiquitous “too busy”)
  • Survey length
  • Surveys that are boring or topics that are boring
  • Repetitive questions or grids
  • The survey experience, including the interviewer’s skills, diction, and attitude (in interviewer assisted methodologies) or technical glitches or difficult to understand vocabulary (in self-administered methodologies)

Peanut Labs conducted research on what respondents like and dislike about survey research, and here is what they found respondents disliked about questionnaires:

  • Too long. 59% of respondents said that the questionnaires are too long – and specifically that they were longer than they were led to believe.
  • Being screened out. 37% of respondents complained about spending a lot of time answering screening questions, only to be told they don’t qualify for the survey.
  • Repetitive questions. 27% of respondents don’t like questionnaires that seem to ask the same question over and over again, even when the questions are worded
  • Boring questionnaires. 17% of respondents said that surveys are boring.

All of these issues can be addressed by designing better questionnaires. As much art as science, researchers must understand that there are human beings on the other side of the question, who have to answer truthfully and thoughtfully. Shouldn’t we do everything we can to make that happen?

How the Questionnaire Fits into the Research Design

Again, one of the most frequent complaints about surveys is that they are too long. The main reason surveys get too long is a lack of discipline in developing the questionnaire. The key is to include only those questions that are needed to answer the research objectives. (For a refresher, review the Infosurv Insider blogs “The Marketing Research Process: 9 Steps to Better Insights” Part One and Part Two).  A questionnaire is not like a Christmas tree – more is not better and adding “just one more” question can lead to a disastrous respondent experience.  Too many researchers still believe that as long as you have a respondent’s attention, you should ask them as many questions as possible.  This attitude leads to high respondent termination rates, respondents providing straight-line responses without any thought, and makes them less likely to respond to future surveys at all.  So the first step in designing an optimal questionnaire is to revisit the first steps in the Research Process.

The next step is to visualize the data elements (aka variables) you will need to collect to answer the research objectives — If you can use secondary research data as part of the information you need, take these into consideration but don’t replicate them in your questionnaire if you don’t need to. Once you have a complete list of data elements, prioritize them. Which data elements are a high priority, such that if you don’t get the information, the research will be a failure? Which data elements are medium priority, and might be considered optional? Which data elements are low priority, which you can live without collecting?  One exercise to help you visualize the key data elements is to write an outline or PowerPoint shell of the report that you want to ultimately complete for your client.  This gives you a better idea of what is really important and what is just nice to know.

The Big Picture of Questionnaire Design

A good questionnaire replicates a funnel. The questions at the beginning of the questionnaire are broad, and easy for respondents to answer (You’re screening questions to find qualified respondents at the beginning of the questionnaire). These first questions, because they are easy to answer, can serve as the warm-up for your respondents to get them started thinking about the survey topic. After the warm-up (and when you have screened out unqualified respondents), you move to the heart of the survey, with more specific questions next. At the end of the questionnaire, we add the most general and easy to answer questions, but those which some respondents may find objectionable (income, gender, ethnicity for consumers and revenue, profitability for businesses).

In the heart of the survey, we move again to the prioritized data elements we identified earlier. Starting with the high priority data elements, put them into the order that will be logical to your audience. Consider whether the order of questions potentially introduces a bias and re-order your list to address that bias. Once you have your list in the correct order, decide the best question format and write the questions for each of the high-priority data elements. Add the medium-priority data elements where the respondent would logically address them, and again inspect the question order for potential bias. Adjust your list, determine the optimal question format and write the questions.

Add page breaks and skip-pattern logic to your nearly final questionnaire. Don’t forget to time the survey to make sure it is reasonable for your audience to complete. Peanut Labs research (noted above) found that 40% of consumers said 10 to 14 minutes is the right length for a survey, 21% said 5 to 9 minutes is about the right length, and 18% of respondents were willing to spend 15 to 19 minutes taking a survey. Note that these are consumers; business respondents are probably even less likely to complete longer surveys.

Now is the time to take a hard, critical look at your questionnaire and ask the following questions:

  1. Will this questionnaire give us the information needed to answer the key business questions?
  2. Are there any questions that we don’t need to meet the research objectives?
  3. Are there questions that seem to ask the same thing over and over?

If the answer to any of these questions indicates a weakness in the questionnaire, now is the time to address it. And finally, think about this: Is the survey interesting for respondents? Good questionnaires have these qualities:

  • They are focused on one topic. If it’s nice to know, don’t ask it!
  • They make a logical connection with the respondent. The respondent understands why they were selected to participate.
  • They respect the respondent. They acknowledge that the respondent must choose to complete the questionnaire with quality answers, and they work hard to allow that to happen.
  • They lead to action. The respondent can easily see how their answers will help the business make decisions.
  • They are engaging. While taking a survey may not be entertaining, they must keep the respondent engaged so that they answer all the questions.

Once you have the best questionnaire, you can develop, program the survey, and quality test it to make sure the logic is correct and there are no typographical errors. Then, pre-test the survey with a subset of your sample respondents. Don’t just use co-workers or family members to test the survey. Nothing can replace the information you gain by testing the survey with “real” respondents. And yes, if you don’t make any changes, you can count these respondents in your completed survey count.

In Part 2 of next week’s blog, the Infosurv Insider will dive deeper into question formats and questionnaire design.

Leave a Reply

Your email address will not be published. Required fields are marked *