Qualtrics Platform
Customer Journey Optimizer
XM Discover
Qualtrics Social Connect

Building ETL Workflows

About Building ETL Workflows

The Qualtrics Workflows platform contains a series of tasks to assist in importing data from third-party destinations into Qualtrics or exporting data from Qualtrics to third-party destinations. These tasks follow the Extract, Transform, Load (ETL) framework. Using ETL tasks, you can create automated and scheduled workflows to bring data from third-party sources into Qualtrics as well as export data from Qualtrics to third-party destinations.

To create an ETL Workflow, you must create 1 or more extractor tasks and 1 or more loader tasks. You are only limited by the overall limit for tasks in 1 workflow.

Qtip: ETL tasks can take up 24 hours to execute. If the task doesn’t succeed within 24 hours, the task will fail.

Qtip: ETL workflows can only run 1 workflow at a time. Simultaneously running workflows may fail.

Attention: An ETL workflow can process multiple files as part of one workflow. Each file may be up to 5GB in size, but the total size of all files combined must be less than or equal to 10GB. Make sure to check the documentation for the tasks you’re using as some tasks have a smaller file size limit.

General Setup for Extractor and Loader Tasks

Qtip: For a general guide on using workflows, see Workflows basic overview.

From the stand-alone Workflows page or the Workflows tab in a project, click Create a workflow.
Select Extract, transform, and load (ETL) data.
ETL workflows usually run on a recurring schedule. Choose a schedule for your workflow. See Scheduled Workflows for more information about setting a workflow’s schedule.
Click Save.
Click Data source (ETL extractor) to choose an extractor task to use. See Available Extractor Tasks for a list of tasks you can use. You can add multiple extractors in 1 workflow.
If you’d like to transform data before you load it, click Data transformation. This step is optional. See Basic Transform Task for more details.
Click Add a data destination to choose a loader task to use. See Available Loader Tasks for a list of tasks you can use. You can add multiple loaders in 1 workflow.
This step is optional, but useful for alerting you if something goes wrong with your workflow. Go to Settings to set up workflow notifications to get notified if your workflow ever fails.
Don’t forget to turn your workflow on.

Qtip: After setting up your workflow, click Run Immediately to to test your ETL workflow and ensure it’s working.

top-right of workflow, next to workflow toggle, there's a run immediately button

Qtip: You can add additional tasks to your ETL workflows. For example, add a web service task to post to a webhook.

Reloading Data

When there’s been a configuration change between ETL tasks that depend on each other, a button will appear to reload all data with the new configuration. Click Reload the data if you’d like to reprocess the old data.

Attention: This will reload all data from the task that was edited. Normally, ETL workflows only process new data since the last run. However, if this option is selected, then the next time the workflow runs, all data since the workflow was created will be processed.

Example: When an extract responses from survey task normally runs, only new responses since the last run will be processed. However, if data is reloaded, then the next time the workflow runs, all responses in the survey will be processed.

Available Extractor Tasks

Here are some of the extractor tasks available at this time:

Extract data from Qualtrics File Service task: Take data you’ve stored to the Qualtrics File Service and save it elsewhere in the platform, such as XM Directory.
Extract data from SFTP files task: Extract data from your SFTP server and import it into Qualtrics.
Extract data from Salesforce task: Extract data from your Salesforce instance and import it into Qualtrics.
Extract data from Google Drive task: Extract data from your Google Drive account and import it into Qualtrics.
Import Salesforce report data task: Extract data from a Salesforce report to import into Qualtrics.
Extract responses from a survey task: Extract data from your Qualtrics surveys and upload it elsewhere, such as an SFTP.
Extract employees from Workday task: Extract employee data from Workday to import into your Qualtrics EX directory.
Extract data from data project task: Extract data from your Qualtrics imported data project and upload it elsewhere, such as your own external database.
Extract employee data from SuccessFactors task: Extract employee data from your SuccessFactors instance and import it into Qualtrics.
Extract recruiting data from SuccessFactors task: Extract your recruiting data from SuccessFactors to import into Qualtrics.
Extract data from Snowflake task: Extract data stored in Snowflake to import into Qualtrics.
Extract data from Amazon S3 task: Extract data stored on Amazon S3 to import into Qualtrics.
Extract run history reports from workflows task: Extract reports about your past workflow runs to import elsewhere, such as an SFTP.
Extract data from tickets task: Extract Qualtrics tickets data to import into another source, such as an SFTP.

Qtip: You can also view a list of extractors using the site menu to the left.

Available Loader Tasks

Here are some of the loader tasks available at this time:

Load B2B account data into XM Directory task: Save imported data to your XM Directory.
Add contacts and transactions to XMD task: Save imported data and transactions to your XM Directory.
Load users into EX directory task: Save imported employee data in your EX directory or EX project.
Load users into CX directory task: Save imported CX user data.
Load data into a data project task: Save imported data from a data extractor task into an imported data project.
Load into a data set task: Save imported data into a dataset to use with a merge task.
Load data into SFTP task: Save imported data to a SFTP server.
Load data to Amazon S3 task: Save imported data to an Amazon S3 bucket.
Load responses to survey task: Save imported data into a Qualtrics survey dataset.
Load to SDS task: Save imported data into a Qualtrics supplemental data source.

Qtip: You can also view a list of loaders using the site menu to the left.

Available Data Transformation Tasks

The following tasks are available to you for transforming the data you process in your ETL workflows:

Basic transform task: Change the format of strings and dates, calculate the difference between dates, perform math operations on numeric fields, and more.
Merge task: Combine multiple datasets into a single dataset.

Troubleshooting Data Extracting and Loading Tasks

Incomplete Workflows

Data extractor and loader tasks must be used together. If you are missing a piece of their setup, the workflows editor will alert you.

Example: In this example, we are missing our data loader task. The workflows editor alerts us that our “import task is currently missing a destination.”

the "missing a destination" error message

Example: In this example, we are missing our data extractor task. The task editor alerts us that “you need to add at least one data source task.”

creating a loader task without an extractor. the task alerts you to set up an extractor first

Workflows Failing

If your tasks are failing or not firing properly, the first place you should look is Workflows reporting & history. This will contain information about every time your workflow fired and the results of that workflow.

In reporting & history, each piece of your workflow will have its own entry, making it easy to pinpoint where things went wrong.

Example: In the below example, we are using an extract data from SFTP files task and a load B2B account data into XM Directory task. We can see that the extractor task failed but the loader task succeeded. This means our extractor task is misconfigured, but our loader task is OK.

the reporting section. the extract task failed but the load task succeeded

After identifying the problem, you can click View under Details to see more information to help you diagnose how to fix the problem. This will pull up the JSON payload for the task. Scroll down to the Task Output section to find any errors.

Example: In this example, we see that the task failed because the connection had invalid parameters for an extract data from SFTP files task.