Building a Platform for Distribution A/B testing

Survey taking has evolved from pencil and paper to more robust options. In today's inundation of noise and technology, it's increasingly challenging to get users to provide feedback about their experiences. One interesting way to boost response rates is to figure out (via A/B testing), what incentivizes individuals to respond to an invitation.

In this post, we will discuss how we enabled our customers to do A/B testing by building a new component on top of our existing distribution system. We reduced our risks with iterative development and increased our ROI with a design that enables future use cases.

Background

With Qualtrics, you can create a distribution via channels like an email, SMS, social, mobile or an anonymous link. A distribution is a delivery to a list of contacts via a specific channel. Distribution A/B testing is a simple way to obtain information from your targeted audience on which distribution channel triggered most engagement. This information can be used to send more effective and customized information to your audience in the future.

Soon, Qualtrics will be launching an updated version of Distribution Reporting which allows users to set embedded data at the distribution level. Embedded data is a form of extra information, in key/value pairs, users may include alongside the information we already store within a given distribution. Distribution embedded data when combined with contact & survey level embedded data, will allow users to run unique A/B tests. Users can use these results to study response patterns and improve user response rates.

Challenge

The feature seems simple enough: set additional info as embedded data in your distribution, run the tests and view reports. However, distribution reporting introduces unique challenges when reporting on distribution events.

First, users can use different channels for distribution, like email, sms, mobile, social, anonymous link, etc. Users can also use different types of distributions: one-way vs interactive two-way for SMS, invite vs reminder for email, anonymous vs targeted links, QR code, etc.

Second, users are frequently interested in the chronological sequence of distributions:  you could be sending out an initial invite, a first reminder or the nth subsequent reminder.

An initial Invite, with 3 consecutive Reminders.

With our current capabilities, it was difficult for a user to make distinctions between distribution events of the same type and channel, like consecutive reminders. This posed a challenge to users who would like to know which reminder incentivized a recipient to open the survey invite email. For the researcher of the audience, having this type of breakdown is crucial in improving audience response rates.

Existing dashboard breaking down response/completion rates against distribution channels.

New dashboard breaking down response/completion rates against a specific sequence of emails (initial invite, vs first reminder email, vs second reminder)

Third challenge is a two-fold problem of managing and using embedded data. Qualtrics sends out a high magnitude of distributions each day. Adding additional data fields to a distribution service can heavily affect performance of key services like caching, database management and reporting.

What do we need to build?

We have a large number of customers using current services and reporting dashboards, so we needed to ensure that there was an automatic migration path to any new available data, and minimal disruptions to existing services & dashboards. We achieved this by building out this solution iteratively. First, we added the ability to store embedded data at the time of distribution. Next, we built a public API to store/retrieve this data for programmatic use, and in the third iteration, we improved the dashboards, onboarded new customers and finally migrated old customers to the new reporting dashboards.

Architecture Additions

Step 1: Storing and retrieving data

Increasing the size and complexity of the data can negatively affect performance in the rest of the stack.  Adding guardrails prevents the system from scaling out of control and provides a basis for our testing limits. For example, using both industry research and Qualtrics customers usage patterns, we decided to limit embedded data to 10 fields (key/value pairs) per each distribution.  We also imposed limits on the content of each field: In the initial MVP, each name can be up to 200 characters long, and values are limited to 1024 characters.

Using above limits, we estimated storage requirements as follows (with 1 byte characters).
5 fields x (15 char key + 200 char value) = ~1 KB per distribution on avg
10 fields x (200 char key + 1024 char value) = ~12 KB per distribution at max

To keep both read and write transaction volume low for MVP, we decided against allowing embedded data to be made available at survey taking time. This allowed us to cut our read/write estimates by a factor of 10. We estimated 10x more reads than writes on average. Let's assume 15k distribution writes per day.  This results in data storage requirements of 150MB per day of ingestion and 2GB of reads.

Our TPS and data volumes are not much but it still clearly pointed us towards a storage management mechanism that delivers rapid read response. To solve this problem, we used a new internal (Qualtrics-developed) object store which is performant and horizontally scales across our global data centers.  It stores objects in a highly scalable cloud datastore, yet caches them in the object’s “home” data center. What this means in practice is that writes are acceptably slow as they must upload to the datastore, yet reads are fast. Based on averages across production, we can expect 85ms on tp99 reads and 850ms on tp99 writes.

Step 2: Building APIs

Both internal and external APIs had to be updated to support the new embedded data field. Some of our key API requirements were:

• RESTful
• Transactional APIs: This API will be consumed by job-workers, integrations & automations, reporting and distribution UIs
• Strong authorization & authentication
• Backwards compatible for public APIs (don’t break existing automations)
• Ease of integration: consumers can access this data from the existing endpoint

Public APIs

APIs for creating distributions were augmented to allow specifying embedded data fields on a distribution. The new subkey is optional, so this change was backwards-compatible.

• POST /API/v3/distributions - https://api.qualtrics.com/v3/reference/create-survey-distribution-2
• POST /API/v3/distributions/:parentDistributionId/reminders - https://api.qualtrics.com/v3/reference/create-reminder-distribution
• POST /API/v3/distributions/:parentDistributionId/thankyous - https://api.qualtrics.com/v3/reference/create-thankyou-distribution

Example call

curl -X POST -H 'X-API-TOKEN: yourapitoken' -H 'Content-Type: application/json' -d '{
    "header": {
      "fromEmail": "apitest@qualtrics.com",
      "fromName": "Api Tester",
      "replyToEmail": "noreply@qualtrics.com", 
      "subject": "Example Subject"
    },
    "message": {
      "messageId":"MS_0Vdgn7nLGSQBlYN",
      "libraryId":"UR_1234567890AbCDe"
    },
    "embeddedData": {
      "emailAuthor":"carmenb",
      "emailTone":"folksy",
      "includedQuestionInSurvey":”true”,
      "reminderStrategy":"multiple reminders"
    },
    "sendDate": "2015-11-17T00:36:42Z"
}' 'https://dc.qualtrics.com/API/v3/distributions/EMD_TEST/thankyous'

Step 3: Improving Reporting

There are many ways to view the distribution data. In the initial version, the feature simply supported queries around initial survey invitations like:

• How many recipients finish the survey after it is delivered to them?
• How many recipients never received the invite?

How can we now use embedded data for more in depth analysis? Let's look at the starting questions we set out to answer:

• What % of my recipients opened the email in a Reminder, vs. the original Invite?
• If an email bounced, what was the type of Bounce? Hard or soft?
• Using distribution embedded data, we can ask: Which of my distributions was the most effective in terms of response rate (A/B testing)?
  • Was it the original invite?  The first reminder? The second reminder?
  • Was it the distribution that used a leading question?  Or the standard invitation?

In order to support user stories like these, we split the cases into 4 work items.

1. Add Reminder Rows to the distribution reporting data set.
2. Add Distribution Embedded Data to the distribution reporting data set.
3. Add BounceType as a data element in the reports.
4. Differentiate between Distribution Channel (Email/SMS/etc.) and Type (Invite/Reminder/etc.) in reporting.

One final key challenge was figuring out how to handle updating existing customer dashboards (lots of them!) to support the new fields. We did this by adding the new schema to the underlying data source. We enabled this schema for new customers before gradually migrating legacy dashboards to the new schema.

New dashboard displaying some custom metrics now possible with Distribution Reporting. Open Rate by Theme/Sequence are specifically enabled by Distribution Embedded Data.

Left: Analyzing Bounce Rate. US has higher bounce rate, but why?
Right: New dashboard with addition of Bounce Type.  Significantly more hard bounces than soft in Germany! Perhaps our email system is incorrectly storing German email addresses.

Next Steps

One simple scenario that can be done now is to use a standard email template for invite, a lead in question for reminders, or different types of lead questions for invites and reminders, then identify which distributions (and strategies) were most effective in terms of response rate.

In the future, we could enable customers to perform more complex A/B testing by combining embedded data in a distribution with embedded data in a survey or answers in a survey response and see how the responses compare in different regions, customer types, etc.

Conclusion

As we built out this feature into our platform, we used the microservices architecture and iterative development paradigms to reduce risks to our existing systems and customers. In addition, this approach helped us enable additional use case where users could combine multiple streams of data to do more complex A/B testing.

As we continue to improve our platform capabilities, our goal remains the same: Provide customers with a robust self-service distribution platform for targeting users and analyzing the success rate of different channels.