Recently, we learned of a company who had completed a survey that included four open-ended questions, each of which had more than 2,000 responses. Let’s do the math: 2,000 open end responses, times four, with an average of 15 words (guesstimated) = way too many responses! During the planning of the survey, they hadn’t considered:
- What they were going to do about the open-ended responses
- How many responses they could expect
- What information each question would provide
- How the information would address the objectives of the study
When the survey was complete, they asked Infosurv to code the responses post hoc. As it turned out upon coding all of the 8,000+ responses, each of the four questions seemed to be garnering similar categories of answers. So, the client could have asked one open-ended question, instead of four. Additionally, many of the responses reiterated information gathered through scaled agree/disagree questions. Bottom line: the client spent a lot of time and money to learn very little, and the respondents had to answer additional, unnecessary open-ended questions.
As a marketing research firm, it is our responsibility to advise clients on questionnaire design and to help them avoid situations such as these. We didn’t design this questionnaire, but if we had, this is what we would have advised the client:
First, determine what you are going to do with the open-ended responses.
As research advisors, it is our responsibility to help our clients understand the implications of each and every question asked in a survey. What will the information be used for? How will it address objectives? With open-ended questions, you need to determine whether or not you need to code the responses into categories for enumeration or whether it is sufficient to just review the verbatim comments. If you aren’t going to code responses, what are you going to do? Is it realistic to think that anyone is going to read all 8,000+ responses? If not, why ask the question?
Upon receiving the results, including a verbatim report of all the open-ended answers, this client decided that they needed to get the responses coded. But for what purpose? Should we only look at the negative comments and suggestions for improvement? Do we need to code the positive responses in any more detail than simply “positive”? Always start with how you are going to use the information.
Structure the Question to Deliver the Information Needed
Determining how you will use the open-ended comments can also help you design open-ended questions that are meaningful and actionable, and that don’t burden the respondents with additional questions.
To ensure that responses deliver the information you need, make sure the questions help structure the responses:
- The question should keep the answers focused. Instead of general questions (What did you like or dislike about this concept?),word the question in a way to elicit more specific responses (How could we improve this concept so you would like it more?)
- Provide directions about how detailed the responses should be. Do you want one or two words, a brief response, one sentence, or a dissertation?
- Is the survey formatted to allow responses the correct length? In online surveys, provide a text box that is the appropriate size for the answers you need.
Choose the Appropriate Type of Open-Ended Question
All open-ended questions are not created equal, so consider which format will best meet your information needs:
- “Other, please specify”: is used at the end of a list of explicit responses. Using this question is a best practice in marketing research to make sure that all respondents have a response that is appropriate for them. (For more about this, see Infosurv Research’s blog on questionnaire design.)
- “Please describe your behavior in the following situation”: can get you more detailed information about consumer behavior in a specific context.
- A closed-ended question, followed by an open-ended question: gets more detail about the “why” of their response to the closed question and is frequently used to explore negative responses to closed questions.
- General reaction questions (e.g. “In your own words…”): These questions give the respondent a chance to paraphrase the concept or situation. The answers may reveal what is most salient to them.
These kinds of questions are valuable tools for the researcher, and can deliver particularly relevant information for decision making.
Don’t Go On a Fishing Expedition
Some researchers advocate including a general open-ended question at the end of every survey. (“Is there anything else you think we should know about this topic and have not asked?”) An open-ended question is meant to allow for complementary or new ideas. Consider whether or not your question is likely to elicit this and whether you can use the data to its fullest. If the questionnaire is already long, or if you believe you have covered your bases to make the decisions you are facing, do you need this question?
Related to this is a caveat not to ask questions that you don’t want to get answers about. The best example of this is in a customer satisfaction survey. If you ask customers an open-ended question such as “What else can we do to improve your satisfaction?”, then you must be prepared to act on their answers. By asking questions like this you are setting certain expectations. If a customer says that you should “improve delivery time to 24 hours or less,” and you are not able or prepared to provide such service, then by asking the question and not acting on it, you have potentially created a new area of dissatisfaction.
Open-ended questions certainly have their place in marketing research, but they don’t take the place of well-designed qualitative research. In online surveys, respondents may not give the detailed responses you need to understand what they are saying. So think ahead, carefully consider each open-ended question, and use them sparingly and judiciously for best results. Your marketing research firm has the expertise and experience to help you with these decisions.