Survey Bias

What are the causes of survey bias? Can you eliminate them?

There are many ports of entry for survey bias, from the population selection method, sampling method, survey design, medium, question and answer wording, the interviewer, and in-particular, the respondents.

Unless you were to survey everyone in your population, it would be nearly impossible to eliminate survey bias – but we can reduce it. Each of the survey sampling methods has its benefits, and unique detractors, here’s a brief explanation.

First, assuming that you’ve chosen the right population you need to decide what your survey sample size is, confidence level, and how you will reach the audience and collect the results. But wait – how do you define the population?

Defining the Population

You could use personas to be very specific, or you may just want to reach a population of “teens in high school who have tried drinking”. Sounds reasonable. But wait – that population definition would rule out anyone who may have dropped out, or who is home-schooled – which would affect your sample. And what about the teens who have tried drinking, but don’t want to respond – they would be underrepresented in your sample.

And, if you are trying to survey the population of “teens who have tried drinking” to see if they are talking to their parents about contraception (vs the consequences of drinking and driving), you may have missed the mark entirely – as this is a small subset of the total population you may want to survey.

There are other forms of Sample Bias:

Sample Collection

Once you’ve established your population, and defined your survey goals, you want to make sure you choose the best method possible for collecting data from your sample. You could choose a from a variety of collection methods, that are either probability sampling or non-probability sampling – and all have pros and cons.

There are different methods to collect data including traditional methods like telephone, in-person, and mail-in, surveying. More modern methods include online surveys via email, web sites, social media, and mobile.

Telephone surveys were very popular for quite some time, but over the years have a developed a number of downsides – particularly for respondents as this article explains – and have shrunken in popularity and effectiveness.

In-person interviews, including focus groups have the benefit of obtaining feedback directly and immediately – however interviewer bias and other social factors such as moderator bias, biased answers, and biased reporting can set in.

Mail-in forms rely on accurately targeting the respondent, and validating that they are the one in the household that completed the survey; it is also subject to non-response bias.

Online surveys have many advantages, but since respondents complete them in private, they are subject to interpretation and therefore question length, wording, style – even format and coloring can affect responses, and survey completion rates.

Social media surveys have inherent issues with the population size and characteristics, don’t allow for more sophisticated survey question types, and can be perceived “fun” or “entertaining” which may not fulfill your survey goals. Surveys on personal networks have the issue of respondents within the network (such as on Facebook) are somehow affiliated with the researcher.

Many online surveys are now distributed or viewed on mobile devices. There can be issues with the survey design itself, and its ability to keep the respondent engaged on the mobile screen. Many surveys are mobile-unfriendly, and even mobile-friendly designs may fail to capture full audience participation.

Sample Size

What’s the right sample size? A lot of that depends on the population size, the ability to reach them, and your budget -however, you need to make sure that the sample size is representative of the population, or the segment of the population (strata) you are trying to gain an understanding from.

Undercoverage happens when not enough members of the population are adequately represented. Non-response bias is when potential survey respondents are unwilling or unable to participate.