Gregling Insight Factory

Customer feedback, survey recruitment, video website testing, and Google Analytics solutions for small businesses, startups, growth hackers, and academics

Survey Design

Protect Your Surveys With The Anti-Dorito Muncher Question

By Leave a Comment

Anti Dorito muncher attention check survey question

Surveys are great for quickly & inexpensively collecting customer feedback to help you rapidly iterate towards a minimum viable product. But like any research tool, there are some minefields you have to watch out for to make sure your results are actually useful.

The unfortunate fact of the matter is there’s no way to tell FOR SURE that the person taking your survey knows what you’re asking, or is just spacing out, checking things randomly and thinking about Doritos, or whatever.

These folks – whom I call, appropriately I think, “Dorito Munchers” – are people who whisk through your survey just checking boxes. This annoying, yet easy to weed out, creature we battle with in market research circles can easily torpedo the best planned out research study.

So aware of this threat, you can lay a trap in the form of an attention check – or what I call an “anti-Dorito muncher question” – to weed them out and keep your survey safe.

Anatomy of the Anti-Dorito Muncher question

This question is a tricky little bugger designed solely to outfox the easily outfoxable.

The way it works is that you just need to lure them in with a simple statement, then create a mini “wall of text”paragraph and then at the end, ask a simple question, the answer of which is in fact wrong. If they were in fact paying any attention at all, they would know that the likely answer is wrong. And if they answer the likely (wrong) answer, then they show they’re not paying attention, and they get disqualified, and your survey remains untainted.

2016-07-26_0055

[otw_shortcode_button href=”https://www.research.net/r/F73NZN2″ size=”large” icon_position=”left” shape=”square” color_class=”otw-red” target=”_blank”]Are you spacing out? Click here to match wits with my attention check question[/otw_shortcode_button]

You don’t have to follow my example verbatim – in fact, I just made it up on the spot. But basically a well-crafted attention check question will have these 3 elements:

  1. The unassuming intro. This is designed to lull your reader into a false sense of security. The Dorito Muncher in particular will see the first sentence, and the large paragraph of text, which will be too much work for them to in fact read, so they’ll skip down to the final sentence. That’s actually what we want. We use their lazy psychology against them.
  2. Level with your survey taker. Here’s where you tell the survey taker what is up. You just tell them that a lot of people often don’t pay attention, which is why you want them to answer the question in a way they normally wouldn’t
  3. This is where you ask the trick question.  In this case, “what color is the sky likely to be?” Obviously if you’re not not reading the question, the answer is obvious – blue, not orange! But in fact when they answer blue incorrectly, you would then disqualify them, and off they would go skulking back into Dorito land.

Advanced anti-dorito muncher question techniques

Now could argue that in fact the above question wouldn’t screen out every Dorito muncher, since as is said the clock is often right twice a day, and they could be so spaced out (and lucky) that they’d answerthat the sky is more likely to be orange correctly. So if you want to add some additional protection, then what you could do is structure the question so that instead of answering wrong, they shouldn’t answer at all. In other words, tell them to skip the question. This would minimize further anyone getting through the cracks.

Then, once you have your surveys collected and it comes time to analyze it, you could simply filter out those that answered the question in your survey results.

Advanced? Not really… but I like the title advanced anti-dorito muncher, so please forgive.

Anyway there you go…an amazing and nearly sure-fire way of protecting your survey from the unaware Dorito Munchers of the world. So until next time…go forth into the world of market research protected from Dorito Munchers…and be sure to practice safe surveying!

 

3 Common Survey Design Errors to Avoid

By Leave a Comment

homePageImage

I work with a lot of clients from all walks of life, most of whom don’t have a formal background in market research, and I see a lot of common recurring errors in the surveys they write that can negatively impact their results and data quality.

So if you’re interested in writing your own survey (but why would you, when you could take advantage of my great survey editing services?), here are 3 common design errors to look out for (and recommended solutions) when creating your own:

Error 1: Not considering that people might not actually like your idea at all

Let’s say you’re a restaurant owner, and you want to find out what people might like you to include on your menu. So a logical choice might be to ask a question like this:

2016-07-24_0347

Seems like a reasonable enough question, right?

The only problem is that the respondent might in fact not like any of these dishes.

Yet – and especially if you’ve made these questions required – the respondent is forced into a troubling dilemma. How to proceed? Likely he or she will simply mark anything to proceed to the next question, and you as the researcher will be left unknowingly with an inaccurate response.

This is known as introducing survey bias.

The solution?

Make sure you always think through the question, “what if my respondent wouldn’t like any of these options?”

In this case, you need to ensure you have a “release valve” for those answers, and one of the best options you can use is, “none of the above.”

In this way, the respondent can more accurately respond with his or her interest, and you as the researcher will be left with more accurate responses.

2016-07-24_0350

Error 2: Including too many answers in multiple choice questions

I recently ran a poll where the individual asked a question about a name preference, and then proceeded to list out 35 different potential answers.

Clearly, the researcher was trying to whittle down a large bevy of options into a more manageable amount, but the problem is that the results could never really be trusted.

That’s because providing too many potential answers introduces another form of survey bias. After all, when there are so many responses, the survey taker is likely to get overwhelmed, and just select anything versus considering them all, which will reduce the reliability & accuracy of the results.

The solution?

Simple – don’t provide too many answer options. Edit down ruthlessly.

Is there a hard and fast rule of how many multiple choice answers to ask? No, but like with all things, try to take a critical look at your question through the eyes of your audience. Would the amount of answers in the question overwhelm you? If so, be ruthless in your editing, and see where you can cut out other options.

In the case of the naming poll, I would have preferred him to try to select his 4-6 favorite items, and then proceed with further iteration in future surveys.

Error 3: Trying to make your survey do too much

As a corollary to the including too many answers, often I just get people asking too many questions with massively long surveys!

Again, the reasoning makes sense – the researcher wants to get as much insight as possible, so they’re going to ask as many questions as possible about a whole range of items on their mind.

However, again as before, your survey takers are human, and when you have a survey that just goes on and on, it’s likely that the respondent may start to lose focus or interest. Again, introducing survey bias.

Solutions?

Your survey isn’t Superman. It doesn’t have to do everything.

Whenever you first sit down to create a survey, you shouldn’t actually start writing questions. What you should do is answer the question, “What is the purpose of my running this survey at all?”

Your purpose should be focused. If you have a ton of things you’re hoping to get out of the survey, then consider breaking your survey into a few smaller, more focused surveys.

Additionally, always try to be as ruthless as possible in editing down and simplifying your survey creation. More is not necessarily better, and can often be worse. In general, the more efficient you can be in getting more information while asking less questions, the better.

Sometimes despite your best efforts you will need to field a longer survey, but in that case there are devices – like including progress bars or breadcrumbs – that can give the survey taker some context which will help with their motivation and focus.

In Conclusion (tl;dr)

Good survey design isn’t rocket science. It just comes down to being aware of a few simple general rules of thumb:

  • Try to put yourself in your respondent’s shoes.
  • Always assume your survey taker might not know, like, or care what you’re doing – and provide a release valve answer like “don’t know,” “not sure,” or “none of the above”
  • Less tends to be more in survey design – try to be as ruthless as possible when simplifying and editing down your questions & answers

Follow these 3 simple rules and you’ll likely be miles ahead of other survey researchers and more importantly, receive more accurate & reliable data in your future surveys.