Defining a Qlik Replicate task

In order to work with Compose, you first need to define a Qlik Replicate task that replicates the source tables from the source endpoint to a landing zone in the data warehouse (defined as the target endpoint in the Replicate task). The landing zone should then be defined as the data source for the Compose project.

For information on which endpoints can be used in a Replicate task that lands data for Compose, see Supported data warehouses.

The steps below highlight the settings that are required when using Qlik Replicate with Compose. For a full description of setting up tasks in Qlik Replicate, please refer to the Qlik Replicate Help.

Prerequisites

When Oracle is defined as the source endpoint in the Replicate task, full supplemental logging should be defined for all source table columns that exist on the target and any source columns referenced in filters, data quality rules, lookups, and expressions.

Limitations and considerations

• Replicate allows you to define global transformations that are applied to source/Change tables during task runtime. The following global transformations, however, should not be defined (as they are not compatible with Compose tasks):

  • Rename Change Table
  • Rename Change Table schema

• The Create target control tables in schema option in the Replicate task settings' Control Table tab is not supported.

• Support for the JSON and XML data types is limited to the Snowflake VARIANT data type. Therefore, apart from the Snowflake VARIANT data type, columns that are usually created with these data types (by the Replicate target endpoint) should be created as STRINGs instead. Therefore, columns that are usually created with these data types (by the Replicate target endpoint) should be created as STRINGs instead. This can be done automatically within Replicate using a data type transformation. For information on which target endpoints support JSON and XML data types as well as instructions on how to create a data type transformation, please refer to the Replicate Help.

• As Compose requires a full-after image to be able to perform Change Processing, the following Replicate source endpoints are not directly supported (as they do not provide a full-after image):

  • SAP HANA (log based)

  • Salesforce

Setting up the task

To define the task:

1. Open Qlik Replicate and in the New Task dialog, do one of the following:

   • To enable Full Load and Change Processing replication, enable the Full Load and Store Changes options (the Apply Changes option should not be enabled).
   • To enable Full Load only replication, enable the Full Load replication option only.

   • To enable Change Processing replication only, make sure that only the Store Changes option is enabled. Note that this option should only be selected if the Full Load tables and data already exist in the landing zone.

   • To enable Change Processing for lookup tables that already exist in the landing zone and are not part of the Compose model, enable the Apply Changes option only. Note that such a task should be defined in addition to the Full Load and Store Changes replication task described above. For more information on updating standalone lookup tables, see Using lookup tables that do not have a task for CDC mapping.
2. Open the Manage Endpoint Connections window and define a source and target endpoint. The target endpoint must be the database where you want Compose to create the data warehouse.
3. Add the endpoints to the Qlik Replicate task and then select which source tables to replicate.

4. This step is not relevant if you selected the Apply Changes or Full Load replication option only. In the Task Settings' Store Change Setting tab, make sure that Store Changes in is set to Change tables.

5. In the Task Settings’ Target Metadata tab, specify a Target table schema name.
6. If a Primary Key in a source table can be updated, it is recommended to turn on the DELETE and INSERT when updating a primary key column option in Replicate's task settings' Change Processing Tuning tab. When this option is turned on, history of the old record will not be preserved in the new record. Note that this option is supported from Replicate November 2022 only.

7. Run the task. Wait for the Full Load replication to complete and then continue the workflow in Compose as described in the Data warehouse project tutorial below and inAdding and managing data warehouse projects .