Setting up data replication in Qlik Talend Cloud

Once you have configured your tenant in Qlik Talend Cloud, you are ready to create connections to your data sources and targets, and then create replication tasks to transition your pipelines from Stitch to Qlik Talend Cloud. The videos in the following sections show you how to connect to common source and target applications. You'll also learn how to replicate data in Qlik Talend Cloud.

Setting up connections to data sources

Connect to any of your Stitch-certified sources using Qlik Talend Cloud connectors. It is recommended that you set up connections before you create a replication task. The following videos show you how to connect to data sources, such as MySQL database, and SaaS applications, such as Pipedrive.

For more information about connecting to data sources, see Connecting to SaaS applications and Connecting to databases.

Setting up connections to targets

The following video shows you how to set up a connection to a target data warehouse such as Snowflake.

For more information, see Setting up connections to targets.

Creating a data replication task

In Qlik Talend Cloud, replication tasks move data between supported data sources and targets. If you have not already set up a connection to the source and target you want to transition, it is recommended that you do this before creating the replication task. For more information about setting up sources and targets, see Setting up connections to data sources and targets. For more information about replicating data, see Replicating data with a Qlik Talend Cloud Starter subscription.

There are two options to consider when choosing how to move your replication tasks from Stitch to Qlik Talend Cloud:

• Using a new destination schema name in Qlik Talend Cloud

• Re-using the existing name of the Stitch destination schema in Qlik Talend Cloud

If you use a new destination schema name, your existing Stitch replication task can continue running against the existing schema. If you choose to re-use the same Stitch destination name, you must pause the Stitch replication task for the connector you want to migrate. Once paused, rename the current destination schema used in Stitch to preserve the historical data.

The following video shows you how to create and run a data replication task.

Do the following:

1. In Stitch, do one of the following: 

   1. If you plan to use a new destination schema name in Qlik Talend Cloud, keep the existing Stitch replication running against the old schema and continue to step 2.

   2. If you plan to reuse the same Stitch destination schema name in Qlik Talend Cloud, do the following:

      • In Stitch, pause the replication for the target connection that you want to migrate.

      • Rename the current destination schema name used in Stitch. This will preserve the historical data under the new name. If preserving historical data is not required, you can remove all tables from the schema, although this is not recommended.

2. In Qlik Talend Cloud, if you have not already done so, create a new connection to the source and target that you are migrating. For more information, see Setting up connections to data sources and targets.

Information noteIn Qlik Talend Cloud, you can reuse the same connection across many replication tasks.

3. In Qlik Talend Cloud, from the launcher menu (), select Data Integration > Home, and then select Replicate data. The Replicate data wizard opens.

4. Follow the steps in the wizard to create the replication task. This includes completing the following tasks:

   • Step 1 - General: Add a Task name, Description, and Project name. Select a Data-Space.

   • Step 2 - Select source connection: Select a data source connection. You can create one if you have not yet already done so.

   • Step 3 - Select datasets: Select the tables and views to include in the replication task.

   • Step 4 - Select target connection: Select the destination data warehouse connection.

   • Step 5 - Settings: Under Replication mode,it is recommended to choose the Apply changes option. This is functionally similar to Stitch's incremental replication. Under Custom schemas, do one of the following:

     1. If you are using a new destination schema name (see step 1a), make sure to choose a schema name that is not actively used in your data warehouse.
        

        

     2. If you are reusing the same Stitch destination schema name, make sure to select the original Stitch destination schema name, not the renamed version that you created in step 1b.

        

   • Step 6 - Summary: To run the task immediately, select Create and run.

5. Once the initial replication is complete, verify the new destination schema against your existing Stitch schema to identify any differences between the two datasets. For more information, see Verifying datasets.

6. Repeat all of the steps in this procedure for each connection you want to migrate.