The Dark Side of Survey Design
Recent YouGov viral poll managed to get 20% of Trump supporters to say they’re against the Emancipation Proclamation, which was Abraham Lincoln’s executive to free all slaves in Confederate states. Major news outlets have used this poll in their stories; for instance, The New York Times stated:
Nearly 20 percent of Mr. Trump’s voters disagreed with Abraham Lincoln’s Emancipation Proclamation, which freed slaves in the Southern states during the Civil War. Only 5 percent of Mr. Rubio’s voters share this view. Mr. Trump’s popularity with white, working-class voters who are more likely than other Republicans to believe that whites are a supreme race and who long for the Confederacy may make him unpopular among leaders in his party.
What was even more surprising was that according to the same poll, 5% of African Americans were against the Emancipation Proclamation. It meant that if the results were accurate, close to 2 million African Americans are against freeing slaves!
YouGov Poll – Demographics BreakDown
There’s a saying among survey design experts: “We can design surveys to make anyone say anything!” While this skill can come in handy if you work for political think tanks/lobbying firms, as commercial market researchers, our aim is to ensure that our surveys are as objective as possible. Let’s take a look at YouGov’s survey and see how this happened.
Bias – For or against executive orders:
The study created a focus on the constitutionality of executive orders (question 47-48), then asked if participants approve or disapprove a list of executive orders.
Note that by asking about constitutionality, participants were now focused on whether the executive orders were constitutional (legal), rather than if they were moral.
Now that participants are focused on the legality/constitutionality of the issue, YouGov asked them if they approve or disapprove a list of executive orders, which “freed all slaves in the states that were in rebellion against the federal government” was among them. While respondents might be morally against slavery, from a constitutional frame of reference – the Proclamation of Emancipation might not be legal since some might argue that the Union had no jurisdiction over the Confederate during the Civil War. Another explanation is that participants disapproved of the executive order because it did not abolish all slavery, only those living in states at war with the Union.
Participant Fatigue:
Participant Fatigue is a well known phenomenon in research, stating that the longer the survey, the more the data quality suffers. The YouGov survey was somewhat long, with 103 questions. By the 49th question, some participants might start to speed through and thus reducing the accuracy of the study.
Some take-aways you can apply to design better surveys:
- Avoid Bias. Ask questions in an objective manner. Pay attention to the the order of the questions to make sure subsequent questions aren’t influenced by precedent questions.
- Ask questions clearly. For instance, question 49 – ask if participants thought if Abraham Lincoln’s executive to free all slaves in Confederate states was constitutional or not, rather than simply “approve” and “disapprove”. By asking questions clearly, you receive clear, actionable insights.
- Keep the survey as short as possible.
Looking to test your ads and increase sales? Spot Trender is a full-service ad testing company – we help our clients test their concepts/storyboards all the way to post-produced spots. Contact us today.
