What DOES a survey "say"?

Suppose for a moment that this blog is incredibly popular, that hundreds of people read it every day. Now, suppose that the blog is all about Justin Bieber – as repulsive as that thought is. I mean, the whole damn thing is Justin this, Justin that, and we all really, really hate Selena Gomez, his current girlfriend. And I mean hate her, with a passion. We do nothing but post blog posts about how hideous she is and how he shouldn’t be with her. We jump on anyone who dares say anything about Justin’s music, calling them all sorts of names and telling them to go to hell. Some of the readers would go as far as to post pranks they’ve played on people who don’t like Justin or have criticized him at some point.

Now, what do you think the results of a survey would be if I asked the following questions:

Do you think that Justin is the most awesome entertainer ever? Yes or no.
Do you think that Selena Gomez is the worst person in the world? Yes or no.

Come on. Think about it a second. After reading what this blog would be about, who do you think would answer the questions and what would their answers be?

See, the thing about surveys is that they are almost always subject to subjective thinking of both the designers of the survey and those who answer it. Any survey is influenced in this way. Just look at some of the questions being asked out there.

Do you think that illegal immigrants, who are breaking the law like criminals do, should be afforded the same rights as law abiding citizens born and raised with the ideals of the United States?

Loading the question much? It would be much more objective if the question was asked this way:

Should illegal immigrants have the same rights as US citizens?

Heck, remove the word “illegal” and replace it with “undocumented”, and you would have a whole other set of responses. It really does matter how you ask the question.

This is why I had no other option but to laugh when an anti-vaccine website posted the results of a study survey that purported to show that unvaccinated children were healthier than their vaccinated counterparts. See, their thesis is that vaccines wreak havoc on the immune system, rendering kids more susceptible to the very diseases that the vaccines are supposed to protect.

Mind you, none of their theses have any remote kind of scientific (non-biased, peer-reviewed, biologically-plausible) evidence behind them. But that’s never stopped the anti-vaccine movement, has it?

Anyway, these bozos posted a survey full of innuendo about vaccines and then asked their readers to answer the survey. Never mind that their readers are – for the most part – anti-vaccine people. Anti-vaccine people who have a severe distrust of the medical establishment, according to all their bashing of the medical system and their prayers that the public health system fall to its knees.

How do you think they answered the questions?

Yeah, and then they said that their web-based survey of their own readers was a “study” and tried to give it some sort of legitimacy. The worst thing is that they did this with a straight face and then got offended that people who know better questioned their results. Really, they did.

So I’m asking one big thing of you, dear readers. When you take a survey, or you are asked to make up your mind based on a survey, read the goddamn questions. Surveys are at best snapshots of public opinion, not scientific fact or any sort of evidence of trends in time… And they are subject to the whims of their creators.

Advertisements