Significance Testing in Dashboard Widgets

Suite
Customer Experience Employee Experience Strategy & Research
Product
Qualtrics
What's on this page

About Significance Testing in Dashboard Widgets

Dashboards can help you understand whether the differences you see over time or between groups are statistically significant, and therefore worthy of driving important business decisions. For example, you may have found yourself asking the following:

  • Did the NPS really go up this month, or is it a small change that’s just noise in the data?
  • Does the Midwest group actually have higher satisfaction scores than the West group?
  • Which of my 5 segments had higher or lower than typical scores on this metric?

With significance testing, you can discover which data changes matter most.

Available Widgets and Metrics

Significance testing is currently available in the following widgets, with the following parameters. We’ll go over these in more detail in the following sections.

Widgets

Qtip: Significance testing is also compatible with any type of dashboard that supports the widgets listed above, including CX, EX, BX, and Results Dashboards.

Metrics

  • Average
  • NPS
  • Top / bottom boxes
  • Subset Ratio
    Qtip: In order to conduct the significance test, the numerator must be a subset of the values selected for the denominator. Additionally, the ratio must be less than one.
  • Custom metrics
Qtip: When testing significance across time periods, the x-axis/y-axis dimension should be a date field. When testing significance across values the x-axis/y-axis dimension should be a non-date field.
Qtip: In some legacy widgets, one type of custom metric can be used with significance testing: proportional custom metrics with a single field as the divisor. Proportional custom metrics follow the general format of (A + B) / C where A, B, and C are different data fields. A / B also works, since there is just a single field as the divisor. When creating custom metrics for this purpose, you can only use counts, as shown below. There cannot be static number in the equation; e.g., (A + 5) / B will not work.
Custom metrics as described above
Custom metrics as described above

However, custom metrics, including proportional custom metrics, are unsupported in the majority of newly created widgets.

Setting Up Line and Bar Charts

Add a line, horizontal bar, or vertical bar widget.
Next to Metrics, click Add.
image of editing a horizontal bar chart widget to have an average metric
image of editing a horizontal bar chart widget to have an average metric
Choose one of the eligible metrics.
Select a field for your metric.
Next to X-Axis, click Add.
image of adding an x axis date dimension to a vertical bars chart
image of adding an x axis date dimension to a vertical bars chart
Add a field of your choice.
Qtip: If you want to compare 2 time periods, select a date field. If you want to compare 2 values, select a non-date field.
Click on your metric.
.
.
Enable Significance testing.

For details on each of these options, see Configuring Significance Testing.

Setting Up Tables

Add a table widget.
Next to Metrics, click Add.
image of a top box / bottom box metric in a table
image of a top box / bottom box metric in a table
Choose one of the eligible metrics.
Next to Rows, click Add.
Add a field of your choice.
.
.
Qtip: If you want to compare 2 time periods, select a date field. If you want to compare 2 values, select a non-date field.
Click on your metric.
Enable Significance testing.

For details on each of these options, see Configuring Significance Testing.

Configuring Significance Testing

Once you’d set up your line chart, bar chart, or table and turned on Significance testing, you’ll have a few options to choose from.

.
.
Decide how you want significance to be determined.
  • Compare current period to a previous time period: Each time period is compared against each previous time period when determining significance of a change. To use this option, your x axis dimension or row must be a date field.
    Example: Is this month’s overall score higher than last month’s? Is this month’s score higher than it was this time last year?
  • Identify particularly high or low values: Most common selection for data that doesn’t involve time or dates.
    Example: Is Brazil’s score higher than other South American countries?
  • Compare current value to another value: Each value is compared against each other value when determining significance of a change. To use this option your x-axis dimension must be a non-date field.
    Example: Is Brazil’s score higher than Venezuela’s score? Is Brazil’s score higher than Colombia’s? Is Venezuela’s higher than Colombia’s?
If you have selected Compare current period to a previous time period, select an offset. This option affects which time periods are used to determine significance. The available options are:
  • Previous period: Compares each time period to the previous time period.
  • 1 year: Compares each time period to the time period one year before.
Example: This widget displays the average CSAT grouped by quarter, with an offset of 1 year. The arrow indicates that the increase in value from Q1 2019 to Q1 2020 was statistically significant.
vertical bar chart widget with a statistically significant value
vertical bar chart widget with a statistically significant value
Select your Confidence Interval. See Understanding Significance Testing for more information on Confidence intervals.

Comparing Significance Across Multiple Dimensions

You can compare significance across multiple dimensions by adding a metric, x-axis, and data series. In order for this to work, you need to make sure that:

  • The x-axis field is a date field.
  • The data series field is a non-date field.
Example: This widget displays the average CSAT for different departments each year. The metric is Average CSAT, the x-axis is the date, and the data series is the department.
We see the widget editing pane with a metric set to average CSAT, x axis set to end date, and data series set to department
We see the widget editing pane with a metric set to average CSAT, x axis set to end date, and data series set to department

Selecting Compare current period to a previous time period compares significance across time periods for each distinct department.
testing significance across time periods for distinct metrics in a vertical bar chart
testing significance across time periods for distinct metrics in a vertical bar chart

Compare current value to another value can also be selected to compare significance across departments within each distinct time period.
testing significance across departments within each time period in a vertical bar chart
testing significance across departments within each time period in a vertical bar chart

Understanding Significance Testing

The Confidence Interval indicates how confident you would like to be that the results generated through the analysis match the general population. Higher confidence levels raise the threshold for a difference to be considered statistically significant, meaning only the clearest differences will be marked as such.

Once you have enabled significance testing, you might notice upwards and downwards arrows in your widget. These arrows indicate statistically significant values.

.
.

You can hover over an arrow to determine why the value is considered significant, and what the confidence interval of that test was.

Example: Here we hover over the blue arrow next to Q1 2019’s CSAT score. The tooltip tells us this value is higher than typical, and the confidence interval for this is 95%.

image of a table with the hover-over tooltip explaining a statistically significant value
image of a table with the hover-over tooltip explaining a statistically significant value

Example: Here we hover over the arrow over the January 2020 – June 2020 NPS score. The tooltip tells us this value for this six month period is lower than the previous six month period (July 2019 – December 2019), and the confidence interval for this is 80%.

.
.

Technical Notes on Significance Testing

When comparing one NPS score to another, regardless of chart type or type of comparison (e.g., over time), the following process is used:

Create a new column of data that recodes NPS scores in the following fashion:
  • Promoters = 100
  • Neutrals = 0
  • Detractors = -100
Run a 2-tailed Welch’s independent samples t-test.

When comparing one top box score to another, regardless of chart type or type of comparison (e.g., over time), the following process is used:

Create a new column of data that recodes raw scores into TRUE or FALSE, depending on whether they meet the top box criteria.
Run a 2-tailed z-test for difference in 2 proportions.

FAQs

Why are my metrics adding up to 99 or 101 instead of 100?

Was this helpful?

The feedback you submit here is used only to help improve this page.

That's great! Thank you for your feedback!

Thank you for your feedback!