Why We Throw Out 30% Of Online Survey Responses

As a survey company, we need people to take surveys. It’s what we do, and it’s kind of a big deal.

At Pollfish, we scour the earth for mobile apps looking to monetize their user base by offering them surveys in exchange for in-app incentives.

Today, we have over 550 million engaged, mobile consumers all over the world ready to take surveys for you. We couldn’t be more proud of the network we have built.

So why, after all this hard work, would we throw out 30% of our online survey responses?

A Brief History Of Survey Panels

For many years, survey companies relied completely on their proprietary audience of survey takers. Through a variety of means, these companies would get people to volunteer to take surveys.

Some companies offered free movie tickets or gift cards. Others were able to find volunteers one way or another. Some companies simply paid for responses.

These proprietary audiences were called panels. These panels are still in existence today at some of the biggest legacy brands in the market research world.

But then, something happened. These survey platforms were too successful.

As market research moved from the large market research agencies to survey platforms on the desktops of any entrepreneur or small business in the world, panel providers struggled to meet demand.

And unfortunately, they got sloppy. They cut corners. They sacrificed the attention to detail that made them trusted vendors in the first place.

How did they fill in the blanks?

The truth is we don’t know. Most panel companies either can’t or won’t say where their respondents come from. These companies likely use some combination of paid respondents and shady sample sub-contractors to fill in the blanks where their panel falls short.

Are respondents being paid? How are they being approached? What type of impression are you leaving with these potential customers once they have taken your survey?

Even more importantly, how do you know if paid respondents are taking your survey seriously? Are they taking the time to give you actual feedback, or are they simply rushing to complete the survey so they can get paid for another?

How We Reduce Fraudulent Completes

Pollfish is different. Through our partnerships with mobile app networks around the world, Pollfish delivers your survey–fully branded and optimized for mobile–to engaged, mobile consumers. In short, paid panel participants are a thing of the past.

Our mobile app network offers one source of survey participants so you always know where your users are coming from.

But with AI fraud protection, we take it one step further.

Paid survey takers are not the only survey respondents who are not giving your survey their full attention.

Because surveys (including ours) come with incentives, many people just want to complete the survey and move on. And they exhibit behaviors that make it pretty easy to discern their motives.

By avoiding survey panels and only using our own responses from real consumers, we can avoid professional survey takers messing with your data quality. But we can’t guarantee users will take your survey seriously.

That’s why we developed AI fraud protection. Using advanced Machine Learning compiled from millions of survey responses, we can automatically remove responses we have good reason to believe may be fraudulent.

Here are just a few things we look for:

Types of fraudulent survey responses

Straightlining: The practice of filling in the same answer for every question in a multiple choice survey. Imagine you are taking a test and you just filled in “A” for every answer. That’s straightlining.

There are ways to protect against this. You can avoid matrix questions or use more open-ended question types.

But paid survey takers are smart–they know they can be penalized for straightlining. So some will offer slight deviations or alternate responses in a predictable manner just to avoid a penalty.

Speeders: These guys speed through every question, answering so fast that it is likely they didn’t read the directions or the question being asked.

Bots: Because panels pay respondents, they incentivize their users to game the system. This comes in the form of bots and fake accounts.

Panels attempt to screen out these users by checking IP addresses. But they often do little beyond that to ensure that their users are all real people.

These are just a few of the problematic survey respondents you should avoid to improve the data quality of your market research. Check out a full list of these bad actors and how to spot them here.

Trust & Transparency

In today’s world, consumer habits and preferences change almost daily. You have to know your customers inside and out in order to keep up.

The speed of today’s market also requires you to move faster than ever before. That means making big decisions based on market research.

Pollfish is laser-focused on bringing trust and transparency to market research, letting you know where your respondents are coming from, and delivering their answers at the speed of your business.

For more information on the Pollfish Survey Methodology, check us out or get in touch with CX by clicking the blue chat icon at the bottom of this page.