The Seven Deadly Sins of Survey Design
Survey design is one of the most important aspects of market research. When you design a survey correctly based on your research objectives, you’ll end up with reliable results. If you hang out with market researchers long enough, you probably hear your fair share of survey design horror stories, and seen many examples of bad survey questions.
1. Assume
Assuming questions means you’re asking a question based on an assumed response. For instance, “Please tell us why you hate political candidate X” is a bad question. It assumes the respondent hates political candidate X, and even if he/she did, you won’t get a complete picture. To get a more objective opinion of candidate X, you could ask “Please tell us your opinion of political candidate X and explain your response”.
2. Bias
One of the most famous examples of a biased question is “please tell us why you think America is the greatest country in the world?”
While many Americans love this question, it’s definitely biased. As a rule of thumb, it is best to avoid taking a stance on anything when creating a survey, especially when it comes to historical and political figures and events. Let the survey participants express their own opinions using a more neutrally worded question. Rather than asking “How heroic is Christopher Columbus?”, ask:
Please tell us how you agree or disagree with the following attributes about Christopher Columbus: [randomized grid question, strongly agree – strongly disagree]
Intelligent
Evil
Adventurous
Unethical
Heroic
Intelligent
Good
Tyrannical
Significant
3. Confusing question with negatives
Similarly to bias, questions containing negatives can also sway opinion, and alienate your participants. It’s an even worse offense if your question contains double negatives. Then, your question is just flat out confusing. Luckily, they are very easy to avoid when replaced with neutral and clear-cut language. Here are some examples of what not to do:
How bad is it to eat fast food?
Are you against non-taxation of tobacco and tobacco-based products?
The first question assumes that fast food is inherently bad and that the respondent agrees. Aim for a more open-ended response by revising the question:
Fast food is ______ to eat on a regular basis.
Very healthy
Healthy
Neither healthy nor unhealthy
Unhealthy
Very healthy
The second question is confusing because it contains double negatives. It is unclear what exactly the question is trying to ask. You can clarify the question to garner thorough and honest responses from your sample:
How do you feel about placing a tax on beverages with a high sugar content? (e.g., soda, juice, flavored iced teas)
In favor
Somewhat in favor
No opinion
Somewhat against
Against
This way, the question is clear, and you can better assess their level of satisfaction or dissatisfaction.
4. All or Nothing / Absolutes
Asking questions with absolutes drastically limits the quality of the responses. It makes the respondent feel that their responses have to be black or white. Give your participants a variety of answer options so that you can have better insights. Yes/No questions are fine for survey screeners if you’re looking for participants with certain characteristics. However, they are best left out of the qualitative portion of the survey. Avoid any word that signifies an absolute, words like “ever” and “always”:
Do you go to the gym every day?
This question is phrased poorly and contains absolutes. It is also a yes or no question, which does not warrant a detailed enough response. Rephrase the question for a better result:
How many days a week do you exercise? (0-1, 2-4, 4-6, every day)
5. Convoluted Questions
You may think that you are saving time by asking a question that serves a dual purpose. However, this mistake can be costly, Whatever you do, do not try to cram two questions into one. While you may think that you’re killing two birds with one stone, it could, in fact, kill your survey.The responses will be fuzzy, and you will not have a clear measure of the results for either question. The solution is simple, just break it up into two questions.
How often do you purchase and eat candy bars?
The above question clearly needs to be broken up into two separate questions. Some participants are going to have different frequencies of purchase and consumption, so it is necessary to have a clear answer for both.
How often do you purchase candy bars? (Never, a few time a year, every few months, once a month, once a week, a few times a week, every day)
How often do you eat candy bars? (Never, a few time a year, every few months, once a month, once a week, a few times a week, every day)
6. Complex language
Lastly, you’ll want to keep your surveys simple and to the point and avoid using industry-specific jargon as much as possible. Even if your survey is industry-specific, or if your sample is made up of lawyers, simple language is still best. You wouldn’t want someone to quit the survey just because they do not understand a question!
How many times a week do you suffer from cephalalgia? (0, 1-2, 3-4, 5-6, daily)
Do you habitually imbibe in alcoholic beverages?
These questions are unnecessarily complex. Survey participants may not know the meaning of the words “cephalalgia” or “imbibe”. If a medical or legal term is absolutely necessary to include for the purposes of the survey, explain the meaning in parentheses. Otherwise, always use the simplest form of the word when possible:
How many times a week do you experience headaches? (0, 1-2, 3-4, 5-6, daily)
Which of the following best describes your drinking habits? (Never, Infrequently, Sometimes, Frequently, Very Frequently)
7. Lengthy
The last sin seems like a no-brainer that often gets overlooked. Many people make the mistake of making their survey too long or including irrelevant questions. Ensure that every single question serves its purpose. The longer your surveys are, the more likely your participants will experience “Participant Fatigue”—it’s a known phenomenon when participants start skipping through your question set because they are lost most of their attention span.
Less is more. If you feel like your survey is getting too long, ask yourself: “will this help answer a key hypothesis?” If the answer is no, then it is best left out of the survey.
Need help creating the perfect market research survey? Spot Trender can help you make positive changes to your brand’s ads? Spot Trender is a full-service ad testing company—we help our clients test their concepts/storyboards all the way to post-produced spots. Contact us now. We’ll work with you to craft powerful messages.
