7 Ways to pretest your survey before you send it

Time is money, and often researchers want to send out their surveys as quickly as possible. But rushing to distribute a survey can result in unforeseen problems with the data collection. Pretesting helps you avoid those issues and raise the quality of your data.

What is a survey pretest?

Survey pretesting means running your survey with a small test group before you do it for real.

If you’re wondering how to pretest a survey, you’ll find a few possible answers to your question. There are multiple ways to pretest, from simple to complex. Depending on the scope and scale of your research, your pretest could be anything from a speedy sense-check to a significant step in your process.

For smaller surveys or polls, it could mean sending the draft questionnaire to a few people inside your company. For a large research study, you might run a pilot test with a larger number of individuals from your sample population, going through every step of the survey process checking for technical and errors and issues.

Whichever level of pretesting you do, it’s always preferable to sending out a survey untested as it can save you time, money and effort.

Read Now: Learn how to improve your digital experience on Qualtrics

Why should you pretest your survey?

It’s always a good idea to test surveys before you send them. Running your survey through a series of tests to check for potential problems can save you a lot of headaches down the road and ensure that you get the data that you want.

Pretesting your survey means you catch any problems with it before you send it out to your participants. If there are questions that don’t make sense, technical glitches or potential sources of bias, you can weed them out early so that they don’t affect your final data.

Pretesting helps you avoid the costly and time-consuming process of scrapping an existing flawed survey and starting over with a new questionnaire and recruiting new respondents.

Pretesting is good for your participants too. It helps make sure their time and effort isn’t wasted on work that won’t lead to useful results.

Prepare for a survey pretest

Before you pretest your survey, it may be worth doing a little planning beforehand, especially if your survey research is complex. Take some time to consider what you want to get out of the survey research and what success will look like. It’s also a good idea to break your survey project down into its component parts and consider which ones you want to pretest and how.

For example, when pretesting a questionnaire, you might focus on:

• Comprehension
  Respondents understand the aim of the survey and the wording of the introduction and questions.
• Logic and flow
  Items in the questionnaire follow a logical order and nothing seems out of place or confusing. At the same time, question order doesn’t set up conditions for bias.
• Acceptability
  None of the questions are likely to cause offense or touch on sensitive subjects in an inappropriate way.
• Length and adherence
  The questionnaire experience isn’t too effortful and most respondents can make it through to the end without losing interest and focus.
• Technical quality
  The survey platform operates smoothly and there are no UX issues.
• Introduction and gaining consent
  The introduction sets out the aim and scope of the research clearly and allows users to give informed consent before they take the survey.

If you’re running a pilot survey, you can set objectives for each of the process, including:

• Branding and appearance
  The questionnaire itself, plus any landing pages, email campaigns or associated branded experiences are consistent and appealing to the audience.
• Questionnaire
  Comprehension, logic, acceptability and the other topics covered above are all satisfactory.
• Data cleaning
  The data cleaning process is clearly set out and working well.
• Data processing
  Data analysis, including crosstab and graphical interpretation plus any statistical testing, follows a clear and consistent process that can be documented and repeated.
• Results reporting
  The reporting format is suitable for the data and shows the results clearly.

Get access to the top downloaded survey templates here: Download Now

Survey testing methods

Here are six different strategies for testing surveys before starting your data collection:

1. Respondent debriefing

This means running your survey on a small number of respondents prior to sending it out to your entire sample.

To get their feedback, you will need to add several evaluation questions to the end of your survey. These can be open-ended or closed-ended questions and usually focus on assessing respondent comprehension and interpretation of survey questions. It should also include overall evaluations of the survey content, time, satisfaction and difficulty.

2. Cognitive interviewing

Cognitive interviews are a good way to really understand what’s going on in the minds of your respondents as they answer your questions. These interviews are typically performed face-to-face with a small sample of 5-15 respondents. As the respondents answer each survey question, they are asked to think aloud, which can include describing their thought processes, providing emotional responses or expressing judgments of their confidence in what each question means.

During a cognitive interview you might discover that certain question wording is ambiguous and needs to be more specific. You might also learn that certain questions make people feel uncomfortable, or that the answer they would like to give isn’t present in a multiple choice list. Cognitive interviews can also help you spot drop-out risks, where the respondents get bored or feel like the survey is too much effort.

3. Expert evaluation

Your survey can be dramatically improved by feedback from two types of experts:

• topic experts who have deep knowledge and expertise about the subject matter of your survey
• survey methodologists that have expertise in how to collect the most accurate data for your research question.

These expert evaluations can help shape the content and form of your survey and result in better data quality and more valuable insights.

4. Focus groups

In the preliminary phases of questionnaire development, it can be very helpful to ask a focus group to discuss your survey. These groups, which are usually semi-structured discussions between 7-15 people led by a moderator, are particularly helpful for clarifying basic concepts in the survey and evaluating perceptions of respondent burden or topic sensitivity.

5. Experiments

Splitting a pretest sample of respondents into groups and testing different variations of your survey design and content can be a powerful way to understand the results you might get when you field your main survey. These experiments are particularly useful for understanding how changes in question wording, questionnaire design, visual layout, question order, and many other methodological factors may influence the data you collect.

6. Pilot surveys

As we’ve described, testing the final version of your survey project from start to finish can be very valuable. To do this, you should recruit a small sample of your target population. We typically recommend that our customers use a sample of about 50 respondents for these pilot studies, or ‘soft launches,’ unless you need to do additional survey testing across different demographics.

A pilot survey can give you a sense of the kind of responses you will receive and any issues that may arise during the real survey period. Pilot studies often serve as a ‘dry run’ and are typically done just before fielding the survey to the entire sample. It’s usually a good idea to include some evaluative questions within the pilot questionnaire, such as respondent perceptions of the length or difficulty of the questionnaire, satisfaction with taking the survey, etc.

7. Data analysis

As well as gathering experiential feedback from your participants, you can look at patterns in their response data to see where confusion, hesitation or disengagement has occurred. Look out for “straight lining” (where every box in a row is ticked regardless of meaning), skipped questions and “don’t know” answers and check these against the feedback respondents have given you to see if there’s a pattern.

Act on what you learn

As with any type of feedback and testing process, the real key to survey pretesting is acting on what you’ve learned. Use the knowledge survey testing has provided to improve your process and survey instrument. And if there are new questions or ideas arising from what you learn, consider a second round of pretesting to make your research project the best it can be.