Onboarding data

The first step of creating a data pipeline in a Qlik Cloud Data Integration data project is onboarding the data. This involves transferring the data from the on-premises data source and storing datasets in read-optimized format. You can update data with continuous change handling, or use scheduled reloads.

You create onboarding in a single operation, but it is performed in two steps.

• Landing the data

  This involves transferring the data continuously from the on-premises data source to a landing area, using a Landing data task.

  Landing data from data sources

• Storing datasets

  This involves reading the initial load of landing data or incremental loads, and applying the data in read-optimized format using a Storage data task.

  Storing datasets

When you have onboarded the data, you can use the stored datasets in several ways.

• You can use the datasets in an analytics app.

• You can create transformations.

• You can create a data mart.

Onboard data

You start onboarding data in a data project. Datasets will be stored in the cloud data warehouse defined in the data project. For more information about data projects, see Creating a data pipeline in a data project.

1. Click Add new and then Onboard data.

2. Add Name and Description for the onboarding.

   Click Next.

3. Select the source connection.

   You can select an existing source connection or create a new data connection to the source.

   For more information, see Connecting to data sources.

   Click Next.

4. Select data to load.

   Click Next.

   Settings is displayed, where you can select update method and history settings.

5. Select which method to use to update data in Update method:

   • Change data capture (CDC)

   • Reload and compare

6. Select if you want to replicate history of previous data in addition to current data in History.

   Click Next when you are ready.

7. Preview the two data tasks that are created to onboard data, and rename them if you prefer.

   Tip noteThe names are used when naming database schemas in the storage data asset. As a schema can only be associated with one task, consider using names that are unique to avoid conflicts with data assets in other data projects using the same data platform.

8. Select if you want to open the landing data asset, open the storage data asset, or return to the data project.

   When you are ready, Click Finish.

The two data tasks are now created. To start replicating data you need to:

• Prepare and run the landing data task.

  For more information, see Landing data from data sources.

• Prepare and run the storage data task.

  For more information, see Storing datasets