Online Panels

About Online Panels

A panel is a group of participants who have been recruited to provide feedback on a research topic. You can define targeting criteria and launch your research study to this audience. For example, let’s say you’re launching a product designed for pet owners and would like to gather feedback on the early concepts. You can use a panel to send your survey to only pet owners, enabling you to quickly collect accurate and relevant feedback without the hassle of finding these individuals yourself.

Preparing a Survey for Online Panels

Follow the below steps to get your survey ready for distribution via an online panel:

1. Make sure your survey is 100% completed in terms of the questions you want to ask, your survey look and feel, etc.
2. Do not configure any screeners or demographic questions that you plan on using with quotas, since these are automatically set up by the online panel.
3. Make sure you’ve added any custom screening questions (for example, asking if a customer has used your product recently).
4. Publish your survey.

Setting Up an Online Panel

When setting up a panel, you can choose from a list of templates and then customize the panel for your study. The high-level process to launch a panel includes:

• Defining your target audience
• Specifying your study’s needs
• Confirming the panel’s feasibility and cost
• Launching the panel

1. Go to the Online panel section of the Distributions tab.
   
2. Click View panel templates.
   Qtip: If you’ve already created a panel, this button will say Create new panel instead.
3. Select one of the panel templates to use as the basis for your panel. You can read more information about the people included in the panel on the right hand side of the screen.
   
   Qtip: You can customize the template in the steps that follow. These customizations include editing and removing quotas, including or excluding screening criteria values, or removing screening questions entirely.
4. Click Next.
   Qtip: You can click Save and exit at any step in the setup to save your panel as a draft.
5. If desired, in the Required Criteria section, choose the country respondents should be from and the language you’d like them to speak.
   
   Attention: The template quotas and questions are based on the default country and language. If you modify the country or language, please make the appropriate adjustments to the remaining quota and question criteria to ensure the panel is valid and feasible.
6. In the following criteria sections (Demographic, Behavioral, or Geographic depending on the template), you can customize the specific quotas and included questions for your panel. The values of the screening questions are stored in your survey dataset as embedded data fields.
   
   Attention: Changes to your panel’s demographics may affect your panel’s cost and feasibility. See Panel Cost and Feasibility for more information.
7. To remove a question from the criteria, deselect the checkbox next to the demographic name.
8. You can select which values for a question are included using the dropdown menu (for example, you can exclude all men if you’d like).
   Qtip: The question text that is shown to respondents is displayed below this menu.
9. If desired, you can adjust the percentage makeup for each demographic group.
   Qtip: Respondents who are screened-out by the screener questions are not recorded and they do not count as auditable responses.
10. Click Remove Quotas if you want to ask the question of your panelists but not require certain quota targets to be met.
    Qtip: If needed, you can click Reset to default template values at the bottom of the page to revert all of your changes.
11. Click Next.
12. By default, your panel details will be prefilled based on your selected template. Review the following fields carefully, and make any necessary changes for your specific study:
    
    • Panel name: Give your panel a name so you can identify it later.
    • Number of responses: The number of responses you’d like collected for your research. You can collect between 50 to 10,000 responses per panel.
      Qtip: When determining this number, you should consider the analyses you want to perform on your collected data. For a 95% confidence interval ±5, the general guidance is ~350 responses per cut of the dataset. For example, if you plan to review male versus female data separately, you would want 700 responses (350 per group). See the Sample Size Calculator for help determining your sample size.
    • Survey length (LOI): The amount of time it takes to complete your survey. This should be between 1 to 60 minutes.
      Qtip: ExpertReview can tell you your survey’s predicted length.
    • Incidence rate (IR): The target percentage of panelists who complete the survey after starting it. See Calculating Incidence Rate for help determining your incidence rate.
    • Time to collect responses: The number of days the panel will be open for responses. This should be between 1 to 20 days.
      Attention: Changing any of these fields except for panel name may affect your panel’s cost and feasibility. See Panel Cost and Feasibility for more information.
      Qtip: If needed, you can click Reset to default template values at the bottom of the page to revert all of your changes.
13. Click Next.
14. Review your panel makeup, cost, and feasibility.
    
    Qtip: If your panel is not feasible or the cost is too high to purchase in the platform, you can reach out to your Qualtrics Research Services representative. If you do not have a research rep or you’re unsure of their contact information, reach out to your Account Team.
15. If you need to edit your panel, click Edit to be brought to that step in the setup.
16. Click Next.
17. Choose how you’d like to launch your panel:
    
    • Soft Launch: Collect 10% of panel responses (up to 50 responses) before collecting more responses. Soft launches take 1 to 3 days to complete.
      Qtip: This is a great option if you want to test your survey length and incidence rate to ensure panel feasibility before purchasing the full panel.
    • Full Launch: Immediately begin collecting all panel responses. You can pause response collection at any time.
18. Click Next.
19. Enter your payment details (credit card only).
    
20. Accept the Terms of Services.
21. Click Purchase Panel.

How Panels Change Your Survey

Once you purchase a panel, your survey will be automatically updated to make sure your panel’s data is collected correctly. These changes happen in your project’s survey flow.

After creating a panel, you’ll notice the following additions to your survey flow:

• A new branch called “Online Samples API – All Fields” at the top of the survey flow, which includes the panel embedded data used to track respondent data from the panel supplier.
• A new security branch at the bottom of the survey flow to ensure panel security.
• When the panel is closed, the security branch will automatically be deleted from the survey flow. The “All Fields” branch will remain so that you can use the embedded data fields in your analysis.
• If you launch another panel on the same survey, any new fields will be added to the existing “All Fields” branch. The security branch will also be re-added while the new panel is active.
  Qtip: Existing fields are not duplicated. Only new fields are added.

Managing Panels

After you’ve requested the panel and your payment is processed, your panel will be created. You can then click on the panel to view more information about the panel’s status, expected end time, number of collected responses, and more.

After selecting the panel, a window will open that displays the following information about your panel:

• Survey: The name of the survey being used for this panel.
• Status: The panel’s status. Possible statuses include:
  • Draft (not launched): The panel has not yet been created because it is not fully set up.
    Qtip: If your panel is a draft, click the Edit panel button to edit it.
  • In progress (soft launch): The soft launch panel is currently open and fielding new responses.
  • In progress (full launch): The fully launched panel is currently open and fielding new responses.
  • Soft launch complete: The panel soft launch was completed and is no longer collecting responses.
  • Full launch complete: The panel full launch was completed and is no longer collecting responses.
  • Paused: The panel is not currently collecting responses but can be resumed.
    Qtip: Pausing a panel does not extend the fielding window. The panel will stop collecting responses 20 days after launching.
  • Collection ended: The panel was not able to collect all responses within 20 days and has stopped collecting responses.
  • Closed: The panel is closed and no new responses can be collected via the panel.
• Launch time: The date and time that the panel was created.
• Expected end time: The requested date and time for the panel to be completed (i.e., when the goal number of responses is collected).
  Qtip: While the expected end time is the target, it is possible that the panel may not complete within this timeframe. The panel will remain open for a maximum of 20 days to attempt to collect all responses.
• Fielding window: The number of days between the launch time and expected end date.
• Distribution Id: The ID of the distribution. This value will also appear on your purchase receipt.
• Total responses: The number of responses collected and the goal number of responses.
• Survey length (LOI): The observed survey length measured in the live panel compared to the predicted survey length.
• Incidence rate (IR): The observed incidence rate measured in the live panel compared to the predicted incidence rate.
  Qtip: If your observed LOI and IR are performing worse than predicted and your panel collection looks unlikely to collect all responses, we recommend closing the panel and creating a new one with updated parameters to get new costs and feasibility.

Full Launching a Panel

If you created a soft launch panel and your test was successful, then you can do a full launch of your panel by clicking the Full launch panel button in the panel information window.

You’ll then be asked to confirm your payment details and accept the Terms of Services. You can also modify the number of requested responses if you choose.

Closing a Panel

You can only have 1 panel open and running at a time. To create a new panel, you must first close your open panel:

1. At the bottom of the panel information window, click Close panel.
   
2. Click Confirm.
   

Panel Cost and Feasibility

Panel cost and feasibility are 2 important metrics when setting up your panel. This section covers both metrics and how customizing your panel can impact these values.

• Cost: Cost is calculated based on your survey length, incidence rate, time to collect responses, selected template, and number of responses. Changing these values will change the panel price.
• Feasibility: Feasibility measures the confidence that your panel will collect all desired responses. Your panel can either be Feasible or Not Feasible depending on your panel setup.
  Qtip: You will not be able to launch your panel if it is “Not Feasible.”

In general, the more complicated your panel audience, the higher your panel will cost and the less likely you will be to find respondents who match your panel criteria. Generally, the following increase the cost and lower the feasibility:

• The more responses purchased (e.g., 5000 responses is harder than 500)
• The longer the survey is (e.g., 50 min survey is harder than a 5 min survey)
• The lower the incidence rate is (e.g., 20% IR is harder than 80% IR)
• The shorter the time to collect is (e.g., 1 day is harder than 10 days)
• The more specific your targeting criteria is (e.g., full time employees in agriculture are harder to find than general US citizens)

Calculating Incidence Rate

The incidence rate is the percentage of people from a target population that meet specific eligibility criteria to participate in your survey and who complete the survey. This section will cover how to estimate an incidence rate for your survey and how the incidence rate for your survey is calculated once you start collecting responses.

Estimating Incidence Rates

Estimating an incidence rate is important because it directly impacts the cost and feasibility of your panel. Follow the below steps to calculate your incidence rate:

1. Define your target population: Start by clearly defining your target population, including specifying relevant demographic characteristics, geographic location, and any behavioral criteria that your audience should exhibit.
   Example: Let’s say we want to survey the general American population who uses smartphones. We’ll continue using this example population for the remaining steps.
2. Specify the eligibility criteria for your target population: Outline the exact criteria that individuals must meet to be eligible for the study survey. These criteria should become the basis for your panel definition and your survey screening questions
   Example: Since we’re conducting a survey about smartphone usage, our eligibility criteria includes owning a smartphone and being within a specific age bracket (18-35, 35-54, and 55+). Additionally, we are only interested in surveying people who live in the United States.
3. Collect relevant data about your audience’s prevalence: Utilize existing data sources to understand the size and characteristics of the target population. This information can come from sources such as government statistics and censuses, industry reports, or internal databases.
   Example: Based on independent research, around 85% of Americans own smartphones. Additionally, US census data shows around 78% of Americans are above 18 years old.
4. Calculate the estimated Incidence rate: Once you have data about the percentage of the population that fits your target, you can multiply your percentages together to find your target incidence rate.
   Example: If 85% of Americans own smartphones and 78% of Americans are above 18 years old, then 66% of Americans are both above 18 and own a smartphone.
5. Use a conservative estimated incidence rate for your panel setup: Online panels are not always distributed in the same ways as a general population and can have biased representation in some areas. For example, online panels tend to have increased representation of 35-54 year old females, and lower representation of harder to reach demographics like 18-24 Hispanic males. It is considered best practice to be conservative in estimating incidence at the start of the study to ensure pricing and feasibility are accurate for your project.
   Example: Since our estimated incidence rate is 66%, we use 60% as our target incidence rate.

Additionally, the following survey design aspects can impact your survey’s incidence rate:

• Survey length: The longer a survey is, the more likely a respondent is to drop off, which decreases your survey’s incidence rate.
  Attention: An attention check is a question designed to make sure the survey respondent is paying attention. Too many attention checks or too strict of attention checks can reduce the number of people who qualify for your survey.
• Complex quotas: When you have a lot of quotas, certain combinations may fill up faster than others. As a result, eligible participants may not qualify because those quotas have already filled up.

Observed Incidence Rates

As panelists respond to your survey, your survey’s incidence rate will be calculated to give you an overview of how respondents are progressing through your survey.

Incidence rate is calculated using the following formula:

Incidence Rate = (Number of Complete Survey Responses) / (Number of Valid Clicks) 

In the above formula, the variables are defined as:

• Number of Complete Survey Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
• Number of Valid Clicks: The total number of times the survey has been clicked, which is calculated by adding together the following values:
  • Complete Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
  • Over-Quota Responses: The number of responses that were removed from the survey because they fulfilled a quota that was already full.
  • Screened Out Responses: The number of responses that were removed from the survey due to failing the screening criteria.
  • Dropped Responses: The number of responses that ended due to 24 hours of inactivity. A dropped respondent can be the result of survey programming errors, redirection issues, or a respondent losing interest in a survey and not completing it.

Creating Custom Screening Questions

If you want to add a screener that is not included in the current targeting options, you can set one up manually in your survey project. Follow the below steps before publishing your survey and setting up the online panel. Keep in mind that more screener questions can cause your incidence rate to be lower (since there are more chances for respondents to be disqualified).

1. In general, screening questions should be placed at the beginning of the survey. We recommend putting screening questions in their own block at the start of your survey.
2. For the screen-out responses, add skip logic to send the respondent to the End of Block if that response is selected.
3. Go to the Survey flow.
   
4. In your screener question block, click Add Below.
5. Select Branch.
6. For the branch logic, build the same logic you did for the question skip logic in step 2.
   
7. Click Add a New Element Here below the branch.
8. Select End of Survey.
9. Click Customize on the end of survey element.
   
10. Select Override Survey Options.
    
11. Enable Redirect to a URL and enter the following URL: https://api.spectrumsurveys.com/surveydone?st=18&transaction_id=${e://Field/transaction_id}
12. Select Do NOT record survey response.
13. Click OK.
14. Click Apply to save your changes.

Viewing Panel Data

You can view your responses by navigating to the Data & Analysis tab of your survey. Responses to screening questions are added to survey responses as embedded data. The embedded data field names use the naming scheme: “_os_api_variable.”

There are a few additional embedded data fields for every survey that uses online panels:

• _os_api_panel_id: The ID of the distribution. This will match your distribution ID for the panel project.
• _os_api_vendor: Third party panel vendor used to source the panel respondents.
• gc: This stands for “good complete.” A value of “1” means the respondent fully completed the survey and will be paid for their time.
• psid: A unique value that the panel supplier assigns to the research participant to anonymously identify them.
• transaction_id: A unique value that the panel supplier assigns to associate the survey session with the research participant.
• pureSpectrumRedirectUrl: Research participants are redirected back to the panel supplier portal on completion of the survey.
• pureSpectrumSignatureValue: A security measure to avoid fraud.

Additional fields are available depending on the demographic criteria used for your panel:

• _os_api_gender_coded: The respondent’s gender. Possible values include:
  • 111 = Male
  • 112 = Female
  • 113 = Prefer not to say
• _os_api_hispanic_coded: Indicates if the respondent is Hispanic. Possible values include:
  • 111 = Yes
  • 112 = No
• _os_api_ethnicity_coded: The respondent’s ethnicity. Possible values include:
  • 111 = White
  • 112 = Black or African American
  • 113 = Asian
  • 114 = Native Hawaiian or Other Pacific Islander
  • 115 = American Indian or Alaska Native
  • 116 = Other
• _os_api_region_coded: The respondent’s region in the United States. Possible values include:
  • 1 = Northeast
  • 2 = Midwest
  • 3 = South
  • 4 = West