Survey Design: Get Off the Grid!

Spread the love

No, we’re not talking about the electric power grid. We’re talking about grid questions, one of the most commonly used question formats in survey research.

The grid question format is simple to use. The response options (such as a level of agreement scale, from strongly agree to disagree strongly) are listed across the top of the grid and each row represents a separate question.  Something like this:

Strongly Agree Somewhat Agree Neither Agree nor Disagree Somewhat Disagree Strongly Disagree

I find this product appealing.

5 4 3 2 1
I find this product unique.

5

4 3 2

1

I plan on buying this product in the next 6 months.

5 4 3 2

1

I would recommend this product to my friends.

5 4 3 2

1

This should be a very efficient way of collecting opinions from respondents and very easy for respondents to answer. After all, you only view the scale once and then read only the question or statement on the left-hand side of the page. Unfortunately, that’s not quite what happens. And with more and more respondents taking surveys on mobile devices, the grid question is difficult to use effectively. The result of using this question type can be bad data. Research on the grid question format has repeatedly revealed the following problems:

  • Higher break-off rates than with other question formats. Survey takers are as much as three times more likely to break off at a grid question than at other question formats.
  • Straight-lining answers or not using the entire scale. Research has shown that many respondents use only three responses on a 5 point scale, resulting in much lower item discrimination.
  • Evidence of not reading and considering each question thoroughly (responding too quickly.) On average, a single question requires 10 to 15 seconds to answer. However, research has shown that grid questions are being answered in 3 to 4 seconds per question.

mobile survey designUsing a grid question format in a mobile survey compounds the problem. The smaller screen makes presenting the grid questions and response categories nearly impossible, requiring the respondent to scroll vertically and horizontally. Accurate response choices may be difficult on small screens. Current research shows that one-third of all online surveys are taken on a mobile device, and that number is rising. But even when the online surveys are optimized for mobile administration, grid questions remain problematic.It is critical that we design surveys that are appropriate to our respondent audience. If your respondents are likely to take the survey on a mobile device, you must design your survey accordingly, and that may mean getting off the grid. Here are some alternatives:

  • Use the one question/one-page rule. Present each question as a self-contained unit, minimizing the need to scroll in any direction.
  • Design shorter surveys. Even if it means doing more surveys on the same topic, keeping your surveys short will improve data quality. Use more advanced analytics (such as correlation analysis and factor analysis) to identify and reduce item overlap.
  • Test your survey on a mobile device. Consider the respondent experience as you go through the survey.
  • Pre-test your survey. Too often the pre-test phase of survey design is forsaken because of cost and time pressures. However, especially as we learn how to optimize surveys for mobile respondents, it is even more important to pre-test your survey to ensure that you get adequate data quality.
  • Leverage other question formats to take the place of the grid. Consider using Simple Conjoint as an alternative.

While the grid question format is a staple of the Marketing Research repertoire, we may have to reconsider our use of this question type. In any online survey, but especially in a mobile survey, question choice has a direct impact on data quality and respondent experience. Challenge yourself to think of alternatives that can deliver the same information, with better results.

Author Image

Lenni Moore

Lenni Moore is the Director of Operations at Infosurv. She’s always been passionate about fostering strong professional relationships. It’s precisely these relationships that allow her to exceed her clients’ expectations because she knows exactly what they want and then leverages her experience to get it for them.