Last week, in our blog “5 Reasons Why Survey Respondents Don’t Tell the Truth”, we wrote about the problem of dishonest respondents. Since the value of marketing research rests solely on data quality, keeping respondents honest is a big deal. In their blog Measuring U reports that the percentage of respondents who give false answers ranges from 1 to 12%, and recommends, based on their own survey, that researchers assume 10% of their respondents are not accurate and should be thrown out.
While it is easy to identify those respondents who are not giving you good information to an open-ended question, you can only disallow their surveys on the back-end of the process. But there are some questions that can be included in online surveys that can help you identify and exclude poor quality data before the respondent completes the survey.
Four types of questions that can help you improve data quality by detecting poor respondents in your online survey:
- Attention Filters. This is a fairly simple approach. Using a large block of instructional text, the instruction on how to answer (“Select 3 for this question.”) is buried in the middle. Alternatively, the question can be buried in a matrix question (“Select “Agree Somewhat” for this question.”) Incorrect answers are programmed to terminate the survey (avoiding paying the incentive.) Using this approach, you can run the risk of eliminating thoughtful respondents who are sincerely trying to answer the question, but who just didn’t read the instructions.
- Trap Questions. These questions are designed to catch those individuals who are speeding through the survey or who are, in fact, cheating on their answers. Trap questions have obviously correct answers, but when they are placed within a matrix question, they can be easy to skip over. Again, if they are answered incorrectly, the survey is programmed to terminate the respondent. Some examples are:
- Please answer very unhappy (Very Unhappy – Very Happy)
- How happy are you with receiving a very large bill from the IRS (Very Unhappy – Very Happy)
- The sun rotates around the earth (Strongly Disagree – Strongly Agree)
- Obama was the first American president (Strongly Disagree – Strongly Agree)
- Reverse Wording. With this trap, you ask the question twice, changing the direction of the scale by asking the question in a positive (or negative) voice. If the respondent is paying attention and answering honestly, the responses should be consistent. If they are not, the respondent is terminated.
- Fake answers. These questions are usually placed at the beginning of the survey, and act as screening questions. By inserting false responses along with true responses, you can not only find respondents who exhibit the behavior you need, but you can eliminate those who are cheating in order to be included in the survey. For example, in a study of respondents with a specific disease, you might ask:
Do you suffer from any of the following:
If the false disease is selected, you know your respondent is just guessing to qualify for the survey. (In case you’re wondering, there is no such thing as Backputter Syndrome!)
- Speed traps. This technique measures the length of time the respondent takes to answer a set of questions. If the respondent is not taking enough time to completely read and answer the questions, they cannot give high-quality responses. Most survey software gives several metrics for the length of time to complete the survey (e.g., total time to complete, time on each page, click counts, etc.) that can help you identify respondents who completed the survey too fast.
For most surveys, adding only one or two trap questions can help you find and eliminate those respondents who are not giving you quality responses. A minor investment to ensure better quality data!