Step 4: Create a Qlik Open Lakehouse project

Create a Qlik Open Lakehouse pipeline project to ingest data from any source. Store your data in Iceberg open table format.

Prerequisites

To create a Qlik Open Lakehouse project, you need:

• A network integration to enable Qlik to provision and manage compute resources on your behalf.

• A lakehouse cluster configured to run the data storage task within your Iceberg project.

• A connection to a data catalog to use as the data target for your project, or the necessary details so you can create a new connection.

Supported tasks

The following tasks are supported in a Qlik Open Lakehouse project:

• Lake landing data task

  Land data in CSV format in S3, from any Qlik-supported source, including high-volume data streams.

• Storage data task

  The Storage data task consumes data landed in the cloud by the Lake landing task. The task write data into Iceberg tables for efficient storage and querying.

• Mirror data task

  Mirror Iceberg tables from your Qlik Open Lakehouse to Snowflake. Users can query data via external tables without needing to migrate data to Snowflake.

Example of creating a Qlik Open Lakehouse project

The following example creates a Qlik Open Lakehouse pipeline project, onboards data, and stores it in Iceberg format tables. This example creates a simple pipeline that you could expand by onboarding more data sources. You could add a Mirror data task to mirror your tables in Snowflake without duplicating data, or use this project as the source for a project that requires transformations in Snowflake.

To create a Qlik Open Lakehouse project, do the following:

1. In Data Integration home, click Create pipeline, and configure it:

• Name: Enter the name for the project.

• Space: Select the space the project will belong to.

• Description: Optionally, enter a description for the project.

2. For Use case, select Data pipeline.

3. Configure the Data platform:

• Data platform: Select Qlik Open Lakehouse from the list.

• Data catalog connection: In the list, select an existing connection or click Create new to add a new data catalog connection.

• Landing target connection: Select the S3 bucket for landing the data or click Create new to add a new bucket location.

• Storage compute cluster: Select the lakehouse cluster that will run the storage task.

4. Create the project.

5. Follow the steps in the onboarding data wizard. For more information, see Onboarding data .