5 Common errors in the research process

Designing a research project takes time, skill and knowledge. If you don’t go into the process with a clear goal and methods, you’ll likely come out with skewed data or an inaccurate picture of what you were trying to accomplish. With Qualtrics survey software, we make the survey creation process easier, but still you may feel overwhelmed with the scope of your research project. This handy guide might help. 

While it’s important to use proper methodology in the research process, it’s equally important to avoid making critical mistakes that could produce inaccurate results. In this article, we’ll list 5 common errors in the research process and tell you how to avoid making them, so you can get the best data possible.

Get your research right every time with our ultimate guide to conducting market research.

1. Population Specification

Population specification errors occur when the researcher does not understand who they should survey. This can be tricky because there are multiple people who might consume the product, but only one who purchases it, or they may miss a segment looking to purchase in the future.

Example: Packaged goods manufacturers often conduct surveys of housewives, because they are easier to contact, and it is assumed they decide what is to be purchased and also do the actual purchasing. In this situation there often is population specification error. The husband may purchase a significant share of the packaged goods, and have significant direct and indirect influence over what is bought. For this reason, excluding husbands from samples may yield results targeted to the wrong audience.

How to avoid this: Understand who purchases your product and why they buy it. It’s important to survey the one making the buying decision so you know how to better reach them.

2. Sampling and Sample Frame Errors

Survey sampling and sample frame errors occur when the wrong subpopulation is used to select a sample, or because of variation in the number or representativeness of the sample that responds, but the resulting sample is not representative of the population concern.

Unfortunately, some element of sampling error is unavoidable, but sometimes, it can be predicted. For instance, in the 1936 presidential election between Roosevelt and Landon, the sample frame was from car registrations and telephone directories. The researchers failed to realize that the majority of people that owned cars and telephones were Republicans, and wrongly predicted a Republican victory.

Example: Suppose that we collected a random sample of 500 people from the general U.S. adult population to gauge their entertainment preferences. Then, upon analysis, found it to be composed of 70% females. This sample would not be representative of the general adult population and would influence the data. The entertainment preferences of females would hold more weight, preventing accurate extrapolation to the US general adult population. Sampling error is affected by the homogeneity of the population being studied and sampled from and by the size of the sample.

How to avoid this: While this cannot be completely avoided, you should have multiple people reviewing your sample to account for an accurate representation of your target population. You can also increase the size of your sample so you get more survey participants.

3. Selection

Selection error is the sampling error for a sample selected by a non-probability method. When respondents choose to self-participate in a study and only those interested respond, you can end up with selection error because there may already be an inherent bias. This can also occur when respondents who are not relevant to the study participate, or when there’s a bias in the way participants are put into groups.

Example: Interviewers conducting a mall intercept study have a natural tendency to select those respondents who are the most accessible and agreeable whenever there is latitude to do so. Such samples often comprise friends and associates who bear some degree of resemblance in characteristics to those of the desired population.

How to avoid this: Selection error can be controlled by going extra lengths to get participation. A typical survey process includes initiating pre-survey contact requesting cooperation, actual surveying, and post-survey follow-up. If a response is not received, a second survey request follows, and perhaps interviews using alternate modes such as telephone or person-to-person.

4. Non-responsive

Nonresponse error can exist when an obtained sample differs from the original selected sample.

This may occur because either the potential respondent was not contacted or they refused to respond. The key factor is the absence of data rather than inaccurate data.

Example: In telephone surveys, some respondents are inaccessible because they are not at home for the initial call or call-backs. Others have moved or are away from home for the period of the survey. Not-at-home respondents are typically younger with no small children, and have a much higher proportion of working wives than households with someone at home. People who have moved or are away for the survey period have a higher geographic mobility than the average of the population. Thus, most surveys can anticipate errors from non-contact of respondents. Online surveys seek to avoid this error through e-mail distribution, thus eliminating not-at-home respondents.

How to avoid this: When collecting responses, ensure your original respondents are participating, and use follow-up surveys and alternates modes of reaching them if they don’t initially respond. You can also use different channels to reach your audience like in person, web surveys, or SMS.

5. Measurement

Measurement error is generated by the measurement process itself, and represents the difference between the information generated and the information wanted by the researcher. Generally, there is always some small level of measurement error due to uncontrollable factors.

Example: A retail store would like to assess customer feedback from at-the-counter purchases. The survey is developed but fails to target those who purchase in the store. Instead, the results are skewed by customers who bought items online.

How to avoid this: Double check all measurements for accuracy and ensure your observers and measurement takes are well trained and understand the parameters of the experiment.

While not all of these errors can be completely avoidable, recognizing them is half the battle. Next time you’re starting a research project, use this blog as a checklist to ensure you’re doing everything you can to avoid these common mistakes.

Also, before you begin your next research project, read How to Define Your Research Question. This is vital to any research project because you can’t begin creating surveys unless you understand the research problem.