Common Survey Errors

About Common Survey Errors

ExpertReview is a feature in the Qualtrics platform that checks the overall quality of your survey before you distribute it, and flags any common survey errors that may exist. It will make suggestions about how to improve your survey to ensure you are collecting the highest quality data possible. Below are possible error messages that you may receive via ExpertReview and ways that you can resolve each one.

Invalid Display Logic

If ExpertReview indicates “invalid display logic,” your display logic may be referencing a question later in the survey, a deleted question, or another deleted item.

Below are some common ways people set up display logic incorrectly, and how to correct this. For a guide on creating display logic from scratch, visit the Display Logic support page.

Order of Logic

The order of your questions is very important when setting up display logic. Display logic can only hide or reveal questions based on previously answered questions. Display logic can’t know what respondents will say to a question they haven’t answered yet. Even if your questions are on the same page, the order is important.

Example: Q1 asks if the respondent has kids. If they say yes, then Q2 will appear asking how many kids they have if in page display logic is enabled. If in page display logic isn’t enabled, Q2 won’t appear until the respondent goes to the next page of the survey.

Example: Q1 asks how many kids the respondent has. However, because of the display logic on the question and the order the questions appear in, respondents will only see Q1 if in page display logic is enabled. Using in page display logic this way should be avoided since display logic should be used to display follow-up questions, not preceding questions.

Deleted Questions

Invalid Logic will appear on your question when you’ve deleted the question being used in the display logic.

Example: Q1, asking if the respondent has kids, has been deleted. Q2, asking how many kids the respondent has, now has an Invalid Logic message on it.

If you try to delete a question being used in display logic, Qualtrics will warn you before letting you proceed.

If you do delete an important question, you may still be able to retrieve it from your trash. Scroll to the very bottom of the survey, expand the trash, click the question, and click Restore in the upper left. If this is the correct question, it will automatically fix the display logic.

If the question isn’t in the Trash, or if the Invalid Logic error remains, you will have to rewrite the display logic. Once you’ve created your new question or decided on a different question you want to base the logic off, click the 3-dot menu and select Edit.

Deleted Quotas

Qualtrics won’t tell you if you’ve deleted a quota being used in logic. First, check to see if there’s a match between what the display logic says and what’s on your quotas page. (Quotas can be found in the survey options.)

If you are still having trouble with the display logic, click the 3-dot menu and select Edit. The field that should display the quota’s name will be empty if the original quota has been deleted. In this case, you will have to recreate the quota and rewrite the display logic.

Example: The Provo Citizens quota has been deleted, so when you go to edit the quota, it says to Select a quota instead of saying Provo Citizens.

Deleted Embedded Data

If you delete embedded data from your survey flow, the system won’t warn you about connected display logic. Make sure the embedded data used in your display logic has a match in the survey flow. Once this is done, you don’t have to edit the display logic, so long as the spelling, punctuation, spacing, and capitalization is the same. For more on adding embedded data to the survey flow, see the linked support page.

Display Logic in Branches

While you can use branching and display logic together, make sure branches don’t unintentionally invalidate your display logic. It is possible to create display logic based on a question that is only shown in 1 survey path. Any respondents who don’t go down this survey path won’t see the subsequent questions with the display logic. ExpertReview will flag these cases for you to review before publishing your survey. Additionally, if you have multiple branches with the same condition, ExpertReview will flag the latter branches. This is because the logic parameter can change in between branches. For example, an embedded data value for a field used in logic may be set and then overwritten in between branches.

Example: Q1 asks respondents their home region and then branches them accordingly.

Q2 asks respondents if they like to ski, but this question is only in 1 region’s block. Later, Q3 has display logic set to display if respondents answer “Yes” to Q2. Since Q2 is located in just 1 region’s block, only respondents from that block will see Q3 even though this question was meant to be asked of all respondents.

Invalid Piped Text

If ExpertReview indicates “invalid piped text,” your piped text may be referencing a question on the same page, a question later in the survey, a deleted question, or another deleted item.

Here are some common ways people set up piped text incorrectly, and how to correct this. For a guide on adding piped text from scratch, see the linked support page.

Order of Piped Text

If you are piping 1 answer or value into another part of the survey, you must make sure the following conditions are met:

1. The answer being piped in and the place where it’s being piped from are on separate pages of the survey.
2. The answer being piped in is asked for before it’s being piped in.

What this means is that Qualtrics can’t pipe in an answer until it knows what it is. This includes submitting the answer so the survey has officially saved it and can use it on future pages. When piping text in, you want to do the following:

1. Create the question whose answer will be used as piped text.
2. Add a page break. This can be done with page breaks, blocks, the look and feel settings, or skip logic that happens in between the questions.
3. Create the question or descriptive text where the answer will be piped.

Example: In the image below, the piped text in Q6 will display the answer from Q5 because there’s a page break between the 2 questions.

Example: In the example below, the piped text in Q6 will display the answer from Q5 if In Page was not selected when setting up the display logic. This is because display logic automatically adds a page break. If In Page was selected, the piped text will be blank since Q5 and Q6 will be shown on the same page.

Deleted Items

If your piped text isn’t displaying a value, it’s possible you’ve deleted the item it’s trying to display data from. Here are a few ways to see if this has happened, and solutions for these issues:

• Every question has an internal ID, or QID, which is used in the piped text. Copy this QID and search for it using the search bar in the upper-right of your survey. The question that appears either has this QID or contains piped text for this question in the question or answer choice text. If the search results are blank, you’ve deleted the question.
  
  
• Quotas have IDs too, but they are harder to search. Quota IDs start with QO_ and are followed by a string of letters and numbers. To see if your quota being used in piped text has been deleted, generate new piped text for the quota(s) you want to check. If the quota ID within the new piped text matches the old piped text, the quota hasn’t been removed.
  
• So long as there are values set in the contact list, you can add deleted embedded data back to the survey flow at anytime to ensure future respondents have their embedded data displayed to them. However, if you are using branch logic to set embedded data values, you may want to make sure your branches are set up correctly, as this may be the reason your piped text isn’t displaying.

End of Survey Scoring Not Set Up

This message means you are trying to display the score at the end of the survey, but there’s no scoring on your survey.

Scoring categories allow you to score your respondents on a group of related questions. For example, let’s say there are 18 questions on biology and 10 questions on chemistry in your survey. Your 18 biology questions can be added together in the Biology scoring category for a total of 18 points max, separate from the Chemistry category. In addition, you can add the same questions to different categories, accounting for overlap.

You can navigate to Scoring by going to Survey options and selecting Scoring.

On this page, you will be able to assign scored values to scoring categories. See the linked page for more details on how to add scores to questions.

Removing Score Reports

Of course, there’s a chance you don’t want to add scoring to your survey at all, or if you do, you don’t want respondents to know their score. Follow the directions to remove the score report.

1. Navigate to Scoring by going to Survey options and selecting Scoring.
   
2. Go to Scoring Options.
   
3. Make sure that next to Show Scoring Summary for Category, nothing is selected.
   
4. Click Save.

Incomplete Translation

If you are still editing a survey or working from a copy of an older survey, there may be times that any translations you’ve added are incomplete. Expert Review should identify the questions missing translations.

This section will discuss how to add missing translations and how to remove unwanted translations. Navigate to translations by going to Survey options and Translations.

Adding a Translation to a Question

Use the arrows to navigate between questions. The circle containing a percentage of completion will be red when a question isn’t translated, or green when it is completely translated.

• Adding your own translations: Click into a question or answer choice text on the right side to enter the text for the translation.
• Automatic translations with Google: Click the red circle. Qualtrics will ask if you’re sure before applying Google Translate to the text.
• Selecting the language to translate: Click the dropdown next to the question number on the right. In the example above, it’s set to Spanish (Latin America). You can add or remove languages from this dropdown using the checkboxes.

Translating Multiple Fields Using a Spreadsheet

You can also use a CSV or TSV to translate multiple fields at once. See the page on Importing a Translation for more information.

Removing Unwanted Translations

Especially when working with a copy of an older survey, you may not want to use the translations in the survey. See the page on Removing a Translation for instructions.

Timing/Metadata Questions Are Alone on the Page

Sometimes, the way your timing or meta info question has been set up may negatively impact your survey.

Common Mistakes with Timing Questions

If you are trying to time a question or set of questions, or if you are trying to limit the time respondents may spend on the page of a survey, you need to make sure that the timing question is on the same page as the questions you are trying to time and that each group of questions you’re trying to time has its own timing question paired with it. Check the Timing Question page for information on adding page breaks.

If you have questions per page customized in your look and feel and you are using timing questions, do not ever set the default number of pages to 1. Each timing question counts as 1 question, and will not properly time the question or questions if it appears on a page alone.

Common Mistakes with Meta Info Questions

The meta info question will not record data until the respondent reaches it and submits the page of the survey it’s on. That means if you add a meta info question to the last page of the survey, respondents who completed only part of the survey won’t have any meta info data.

In order to ensure that data for the question is recorded, be sure to place the meta info question in a block with other questions that are displayed! We typically recommend placing it as the first question in your first block of questions. This can help you avoid any logic that may cause the question to be skipped. Since the meta info question is not displayed to the respondents, they will not know the difference.

You can move a question to the top in 2 ways:

• Clicking, dragging, and dropping the question to the desired location.
• Open the question editing menu and click Move question. You will then be able to choose the block where you want the question to be moved and specify that it should go to the top of the block.

Example: Here, we put it at the top of the Default Question Block because that’s the first block in the survey.

More Than 1 Timing / Meta Info Per Page

There should never be more than 1 timing question per page. There should also never be more than 1 meta info question per page. (You can add 1 of each to the same page, however.)

For guidance on the many ways you can add page breaks, see the Page Breaks support page.