Panels Project

FAQs

People Involved & Coverage Hours

• Why have so many different people reached out to me? I prefer to work with one point of contact.

  We have a few different roles on our team, and each person is responsible for something specific. We specialize in different areas to ensure that you receive the highest quality of service and deliverables as possible. Our team is made up of Account Executives, Operations Specialists, Project Managers, and Support Project Managers. If you prefer to work with only one or two people, please let us know and we can arrange accordingly!

• What will the Account Executive or Operations Specialist help with?

  Your primary point of contact for all new studies is your Account Executive. He or she will be the one involved in the early stages, helping with the bid, contract, scoping, etc. This will also be the person you reach out to after your study is complete and you want to run another study.
  Some of the reps have someone who directly works with them to assist them with their large workloads, which is why you may see another name pop up during those early stages.
  These are our Operations Specialists, and they work closely with your Account Executive to ensure they’re on the same page and that everyone is up to date.
  All questions regarding feasibility, pricing, etc. will be handled by your Account Executive (or will be decided by your Account Executive and messaged to you by your Project Manager).

• What will the Project Manager or Support Project Manager help with?

  When you are ready to begin work on the study, you’ll be looped in with a Project Manager. Your project manager will be responsible for taking the project through four Project Stages. Your Project Manager will be your main point of contact for the specific study at hand from the time he or she is looped in with you until the time the study is complete.
  All questions about data quality, timeline, incidence rate, etc. will be handled by your Project Manager.
  We also have a Project Manager here providing coverage from 5-10pm MST Monday through Thursday, so you may see “Panels Projects” jumping in to assist while your primary project manager is out of the office for the day.

• What coverage can I expect while my study is in field?

  We provide full coverage during normal business hours – the majority of our Project Managers are based in Utah, USA and work until 5pm MT. No other weekend or holiday coverage is provided.
  
  APAC-JAPAN (APJ) SPECIFIC:
  Our project management team based in Sydney, Australia work until 5pm AEST. In terms of launching a new project, cut off times are: survey link delivered by no later than 3pm AEST (Monday-Thursday) or 12pm AEST (Friday).

• What do I do if I need urgent help outside of normal business hours and/or coverage hours?

  If you have an urgent need outside of our standard hours, please send an ATTN email (by adding ATTN before the existing subject line on the email thread) to your Project Manager, Account Executive, and panelsprojects@qualtrics.com. One of us will respond as soon as we’re able. We try to check in periodically outside of business hours, but our response may be delayed until business hours resume.

Fielding Process

• My Account Executiveresentative looped me in with a project manager. Is my study live?

  This means that a project manager is assigned to program and launch your study, but it does not mean it’s live yet. Your project manager will reach out to get your final confirmation of full project specs before moving forward. He or she will then program the survey as described in the FAQs and will test the study before launching.
  Most studies are launched within 24 business hours of the time you are looped in with the project manager, but that also depends on your confirmation of the specs and the complexity of the setup. Without your confirmation, we can’t launch – so a faster confirmation usually equals a faster launch!
  We try to get all studies launched within 4-8 business hours of your final confirmation to the project manager, but please keep in mind that depending on workload, capacity, complexity of your study, etc., it’s very possible we’ll need more time (still aiming to fit in the 24-business-hour window).

• What process does Qualtrics use? Now that a project manager has been assigned, what can I expect?

  See the Project Stages page for an overview of our process, and click into the specific pages to see more details if needed.

• Is soft launch data test data or does it count toward my total?

  Soft launch data comes from valid participants who qualify for your study, so those participants do count toward the requested number of completes for your overall project. If there is an error on our end (people got in who shouldn’t have, we launched when you told us not to, etc.), we’ll be happy to remove and replace the soft launch data. However, if you make any changes to the survey (including minor updates, logic fixes, etc.) or if you simply don’t want that data, you will be responsible for either keeping the soft launch data in your data set or paying to replace it.

• What am I supposed to look for in the data you send me after the soft launch?

  The soft launch is to make sure everything looks good at a high level. Are all your questions showing? Is the screening logic working properly? Do you see an unusually large amount of invalid data? These are the basic questions we want to answer before moving forward.
  To answer these questions, we first recommend utilizing this checklist to know what to look for. If you’re unsure how to read the results (in Qualtrics or in Excel), please Contact Support and they can assist you!

• How long should I expect my study to be in field?

  A simple study (high incidence, few screeners) will generally take about 3-5 business days. More targeted and/or complicated studies will take more time, sometimes about 7 business days and sometimes a few weeks. We will let you know the amount of time we anticipate your specific study will take. This will sometimes change depending on performance in field (e.g., a lower incidence than expected will increase field time), but we’ll always do all we can to complete your study by your deadline.
  There are a few key times when you can help the process. The first is when we send the initial specs. We require your confirmation of the project details before we set up and launch your study – just to make sure we’re on the same page. It’s the last check before data collection, so it’s crucial that we work together to make sure nothing slips through the cracks. A quick reply to that email and to any follow-up questions from it will generally mean a quicker launch.
  The next key step is when we send the soft launch data. The quicker you review that and get back to us with either (a) your questions or (b) confirmation that we can proceed, the quicker we can resume fielding, and the quicker fielding will complete.
  Ensuring that you are responsive throughout the process is the best way you can help move things along.

• I want to sample incrementally – 100 at a time for the 1000 completes. Can you do this?

  Absolutely! Let us know the sampling plan you’d like, and we can discuss the options (including whether that will change costs and if it’s possible).

• Why did you collect more responses than I requested?

  On most studies, we oversample by 5 percent to provide you with proactive replacements should you find any invalid data. These responses are provided to you free to charge. However, at any time – especially if we are struggling for completes, if we’re on a tight timeline, etc. – we reserve the right to not provide these responses and instead collect only what you requested. That being said, if we don’t oversample and you find invalid data (defined as gibberish/nonsensical responses or straighlining throughout the study), we will be happy to throw it out and replace it for you free of charge.
  We do not oversample on B2B or medical studies.

Survey Setup

• What is the project manager going to change?

  Your Project Manager will make some changes in your survey before collecting data. These changes are made to ensure that we collect the correct data and can keep track of which participants need to receive compensation.
  Some of the things your Project Manager may change include:

  • Skip Logic
  • Embedded Data and Screening Logic
  • Redirects
  • Quotas
  • Reports
  • Quality Measures Within Your Survey

• What is your QC/QA process?

  Unfortunately, our team does not do any type of full quality assurance, quality testing, survey review, etc., as our Project Managers are not programmers or survey methodologists. That said, we do offer this as an additional service and can bid this out for you if you’d like. Please reach out to your Account Executive for more information and pricing.
  Note that your Project manager will be conducting some tests to ensure that the survey programming we do (see earlier sections) is correct, but he/she will not be reviewing the rest of the setup. Sometimes the Project Manager will notice something that looks incorrect and will bring it to your attention, but this is not part of our standard process.
  Please be sure that your survey is finalized and has been reviewed on your end before launch. Utilizing the “Survey Preview” option within qualtrics is a great way to test.

• Can I unlock my survey to make changes?

  We recommend checking with us before making any changes to the survey once we are part of the process, as some changes can impact our programming or already collected data.
  Any changes such as updating question wording, adding answer choices, renumbering questions, updating coded values, etc. are always allowed and won’t affect our setup or existing data.
  Changes like deleting a question or answer choice or changing a question type will almost always delete the data tied to that question/answer choice. We usually recommend against such changes, but if you’re okay with potentially losing the data there, those changes are okay.
  Other changes that impact the group you are sampling to, such as adding new screening questions, should always be checked with your project manager, as those can affect study costs, feasibility, timeline, etc.

• My study (copied from a past project, live in field, or completed) won’t let me test. It just kicks me out! Why is this happening?

  There are a few things that may be preventing testing. The main reason why you’ll be kicked out immediately is if we’ve collected the needed responses and the quota is full. Another reason is if you have “Prevent ballot box stuffing” checked in the Survey Options. This places a cookie on your browser when you take the study and tells the survey that you can’t take it again after that. This security measure will give you a message saying “You have already taken this study.” and will prevent you from testing but can definitely be removed if needed.
  We also add programming to screen people out if they don’t meet the screening criteria or fail an attention check. Make sure when testing on a live study that you’re responding as a participant who would qualify, and you should then be able to get through.
  Let your project manager know if you can’t test, and he/she can help troubleshoot what’s going on and can open it back up for testing.
  If you run a study with us and plan to use it later for your own responses, we recommend making a copy of the study so that one is used for our sample and one is used for yours. This can prevent the issues mentioned here.

Sampling

• What are your sampling methods?

  Our sampling methods vary depending on the sample source we are utilizing for your study. Some studies are sent out using specific invites to respondents’ email inboxes, and others are deployed using more of a dashboard-type system where we show respondents lists of studies that they might qualify for, based on targetable items.

• Do you deploy invitations that are nationally representative?

  By default, we do not sample with nationally representative invites or any other specific breaks, including quotas. Online panels are not probability samples and are not made with national representation. Some groups, such as females, middle-aged respondents, Caucasians, etc., tend to be highly represented and often respond quicker to surveys than other groups.
  We may be able to include some demographic quotas in your survey to ensure that the final respondents are representative of the numbers you’d see nationwide, but this can often impact costs, feasibility, timeline, etc. Please speak with your Account Executive to get more information here.

• Why does the soft launch seem to not have the quotas implemented that I requested?

  Quotas are set on the overall study’s target and not on the soft launch target. We do not want to turn away respondents who we will need later on, so we prefer to not set quotas on the soft launch. Let us know if this is needed though, and we can discuss.

• My study is complete and I need some extra information – response rate, number of invitations, size of the panel, a generic example invite, respondent incentives, etc. Can I get that?

  Depending on the sample source we use on your study, some information may be available after the study and other information will be unavailable. We can always let you know incidence rate and the number of terminates on each terminate question, and we also have example invites. However, with new sampling methods that don’t use specific invites, we often will not be able to provide response rate, number of invitations, etc. We also are never able to share the size of our panels.
  Please don’t hesitate to let your project manager know what items you’re looking for, and he/she will let you know what we can and can’t provide for your specific study.

Data & Analysis

• What are partials? Why would I record them? Why not? What’s an annual license?

  Partial responses come in three types:

  1. People who drop out of the survey (exit their window or tab)
  2. People who are screened out of the survey
  3. People who qualify for a quota that is full (i.e., over quota)

  These responses are not full responses and contain responses up to the point where that person was screened out.
  Partials will count toward your annual license but will not count toward your project total (i.e., the number of requested completes will not be impacted by recording partials). To view the response allotment for your annual license, select the silhouette in the top right corner of Qualtrics, choose “account settings” then select “account usage”.
  Some people keep partial responses to be able to analyze deeper incidence rates (e.g., maybe you screened out on people with an income under $30k or over $100k and want to know the number of married people who had too low of an income versus married people who had too high of an income).
  Whether or not you record partials in your account, we collect the number of people who screen out on each screening question but do not record their specific answer choice to that question and also do not record any other answer choices or data on those respondents. So, if you need detailed information about incidence rate or those who didn’t qualify, we recommend recording partials.
  Others record partials to find completion rates, because their account license is unlimited, etc.
  Most people do not record terminates because they’re interested only in full completes and/or don’t want partials to count against their annual license. Many studies have thousands of partial responses/terminates, so these can clutter your data as well.

• I recorded partial responses and am not sure how to easily identify the valid completes versus the partial data. Can you help?

  We use a coding system to easily check the status of each participant. These are set as embedded data tags for each participant. The name of the embedded data field is lowercase gc.

  Here’s the coding system we use:

  • gc=1: valid complete
  • gc=2: terminated response
  • gc=3: over quota response
  • gc=4: quality terminate

  If the gc value is empty, the response is likely from someone who dropped out of the survey. Occasionally over quota responses may come through with an empty gc value as well.
  We recommend simply filtering by “If Embedded Data gc is equal to 1” to see/download only the valid completes.
  Alternatively, if you want to view only the terminated responses to see how they answered the questions they did see, we recommend using the filter “If Embedded Data gc is equal to 2”.

• Why do the values in the Excel file not make sense? How do I read them & why are some not intuitive?

  Each answer choice, with the exception of open-ended responses, automatically receives a coded value. Those are, by default, set to be sequential in the order in which the answer choices are created for that specific question. Often times when questions or answer choices are edited, copied, etc., the coded values will change.
  Please reference our recode values support page to learn more about how to view and edit coded values. Any changes to coded values are retroactive and can be made before, during, or after data collection.
  Another quick way to view coded values in a “key” or “legend” format for the full survey is to export the survey to Word format, which will have the coded values in parentheses next to each respective answer choice.

• Will you be scrubbing my data for me?

  Standard projects do not include a data scrub. We offer scrubbing as an additional service, though! Please reach out to your Account Executiveresentative for more information and pricing on data scrubbing.
  We recommend adding approved quality checks to help decrease the amount of invalid data you find, and we also generally include about 5 percent extra responses to act as proactive replacements should you find any invalid data. Ask your project manager about approved quality checks.
  That said, we’re happy to remove and replace clearly invalid responses that you identify and send over to us. This includes people who straightline throughout the survey or who enter gibberish responses.
  Note that we do not provide extra responses on B2B or medical studies but are still happy to replace bad data. We also are not obligated to provide these extra responses if we are working on a tight timeline, have a low incidence rate, do not have feasibility for extras, etc.

• Can I delete the invalid responses that I found?

  If it’s a small number of responses (i.e., 10 percent or less of the current number of responses), absolutely! Just send us a list of the respondents you removed – the best format to do this is to highlight an .xlsx file so that we have all the necessary information about each removed respondent.
  Please remember to select the “Decrement quotas” box when deleting responses so that the quotas will remain correct. We can delete them on our end, if you want, to ensure all the correct settings are applied.
  If you are throwing out a large percentage of the data (i.e., over 10 percent), we will need to review and approve the removal of those responses so that we can ensure we are doing all we can to keep data quality high moving forward. We also need to determine if we also would consider those responses invalid.
  Please ask your project manager if you have further questions about invalid responses and/or what can be removed.

• My study is complete – can you help me with reporting and analysis?

  We can certainly provide analysis services and help at an extra fee. Please reach out to your Account Executiveresentative for more information and pricing.
  Our project managers are data-collection experts but do not handle reporting or analysis beyond basic monitoring reports. Reporting and analysis should be handled on your end, but we have a lot of great resources from our support team to help with this. You can search for what you need on our Support Site or contact support by clicking “Help & Feedback” in the top right of your account.

• Can you download the data for me? How can I do that myself? What formats are available?

  The data can be downloaded on your end right from your Qualtrics account! Please view our support page on downloading data to learn more about how to do this and what formats are available.