It’s a spooky thought. Lurking in your survey data are nuances that haunt you every time you field a study. As you sift through your data they appear then disappear as cryptic texts, so puzzling you may not even be aware of what they mean. Often ignored or thrown out, these seemingly minute occurrences can signal something much more sinister about your research design…
As you dry your brow, take a deep breath and allay your fears – we’re only talking about survey dropouts! Dropouts occur when respondents become disengaged and leave an online survey before completing it. Removed from the pungent Halloween humor, dropouts are a very prevalent occurrence in survey data. The reason is because these disengaged respondents may be telling you something in particular about your survey by their very actions. So are dropouts just a meddling trick to be ignored, or can they be a valuable treat in disguise?
In some respects dropouts are tricks, that have to be managed seriously. Dropouts can complicate a survey administration by requiring a larger sample size. As more respondents abandon a study, more sample is needed to try and replace the incomplete answer sets. This may not be a huge problem if you’re using a sample provider with a large sampling frame, but if your sample size is small to begin with, you may end up losing out on the data you need to complete your study.
On average, about 10% of all respondents will drop out before the survey is completed, even if it takes only a few minutes. Our in-house analysis of surveys taken over the past year showed 7% of our customer survey respondents and 8% of market survey respondents dropped out by 15 minutes into the survey, on average. These averages reflect a range of dropout rates, from different length surveys. Longer surveys can see higher drop out rates (as many as 30%).
This data not only sheds light on the rate at which respondents drop out of a survey but it also signals to survey designers that once the survey passes a certain length of time, respondents start to disengage and abandon the survey. This is very important because while abandonment tends to plateau at a certain level, it certainly indicates that respondents are not engaged. As respondents become disengaged with a survey, central tendency and habituation can take effect – resulting in poor answers. To alleviate this effect there are some best practices researchers can implement:
- Shorten the survey – There is a positive correlation between survey length and abandonment rates. The longer the questionnaire, the higher the abandonment rate. When designing a survey, take into the length of time it will take the respondent to complete it, as well as the amount of pages, length of text blocks, etc. Shorter surveys have also been shown to deliver higher quality feedback. That being said, survey length should be of chief concern when looking to decrease dropout.
- Limit the amount of complex and open ended questions. This will reduce the potential for respondent fatigue while also decreasing the complexity and time necessary to answer each question.
- Implement incentives that increase people’s willingness to complete the survey. Incentives should be relevant to the respondents’ demographic while sufficiently compensating them for their time and feedback.
- Our analysis found the largest portion of dropouts are on the 1st page. We saw as many as almost 4% of respondents drop out of our surveys before even going into it. This indicates that something on the 1st page isn’t connecting with the respondent. There are a few suggestions to guard against dropouts here:
- Engage respondents with a compelling introduction
- Clearly explain the purpose of the study, why their feedback is important, and what’s expected of them.
- Make sure they’re fully aware of the length of the survey before they begin. State something like, “This survey contains XX questions and should take XX minutes to complete.”
- Mention any incentive and remind them they must complete survey to receive it.
- Use progress bars to keep the respondent posted on how they’re doing. There are mixed views on the effect progress bars have on the completion rates, but we believe the progress bar is helpful to the respondent and shows them that the designer values their cooperation.
- Implement a countdown timer that sets a time limit on the survey. This has been shown to reduce dropout rates given that respondents have a real-time metric that urges their completion without unneeded breaks, etc.
- Make your research more engaging! We’ve found that drop-out rates are much lower for studies that utilize prediction markets, gamification, and crowd intelligence. If respondents are engaged with the research, they are likely to provide more feedback and give you more of their time.
While losing as much as 10% of your survey respondents may not be scary at first thought, there are some complexities dropout may present in addition to signaling some design changes to improve your survey. Infosurv’s skilled researchers take survey length and other factors into consideration when designing a questionnaire to deliver the optimal response rate and high quality feedback. If you’re interested in fielding a survey that gives you treats without the hidden tricks, contact Infosurv today at [email protected] or call (888) 262-3186 for your research consultation.
