What is Qlik Replicate?

Qlik Replicate is a simple, powerful, easy-to-implement solution that provides replication between various endpoints. Replicate lets you:

• Load data efficiently and quickly to operational data stores/warehouses
• Create copies of production endpoints
• Distribute data across endpoints

Replicate is designed to scale and support large scale enterprise data replication scenarios with a scalable multi-server, multi-task, and multi-threaded architecture.

Replicate consists of a Web-based console and a replication server to replicate data across both heterogeneous and homogeneous data sources. It provides users with instant visibility into current and historical tasks, status, performance, and resource usage information.

Replicate can execute replication tasks between a large number of enterprise endpoints including Oracle, Microsoft SQL Server, and IBM DB2. It uses a "Click-2-Replicate" design that simplifies the replication process by automating the steps required to build a replication solution.

When you set up a task in Replicate, you can select any of the following:

• Full Load Replication: Creates files or tables on the target endpoint, automatically defines the metadata that is required on the target, and populates the tables with data from the source
• Change Processing, also called Change Data Capture (CDC): Captures changes in the source data or metadata as they occur and applies them to the target endpoint as soon as possible, in near-real time
• Store Changes: Captures changes in the source data or metadata as they occur but does not apply them to the target

Replication is log based, which means that it reads only the changes. This reduces the impact on the source endpoints.

System architecture

The following diagram shows the basic architecture of Qlik Replicate.

In the initial load process, Qlik Replicate reads a filtered stream of rows (with relevant columns only) and passes them to the transformation process for further filtering and subsequent writing to the target endpoint (in the expected output format).

The CDC process obtains a stream of filtered events or changes in data or metadata from the transaction log file. It then buffers all changes for a given transaction into a single unit before forwarding them to the target when the transaction is committed. During the initial load process, CDC also buffers all changes that occur within a transaction until all affected tables have been loaded.