Azure Data Lake Storage

Azure Data Lake Storage can be used as:

• A cloud staging area when using Databricks, Microsoft Fabric or Azure Synapse Analytics as a data pipeline target. The cloud staging area is where data and changes are staged before being applied and stored.

• A target in a replication task.

  • For an explanation of how to replicate data to Azure Data Lake Storage with Standard, Premium, and Enterprise subscriptions, see Landing data in a data lake with a Standard, Premium, or Enterprise subscription.
  • For an explanation of how to replicate data to Azure Data Lake Storage with the Starter subscription, see Replicating data with a Qlik Talend Cloud Starter subscription.

Limitations and considerations

The following limitations apply:

• Full LOB Mode is not supported.
• Database names, schema names, or table names containing slash (/) or backslash (\) characters are not supported.

Storage permissions

The Azure Active Directory tenant specified in the connector settings must be granted the following ADLS Gen2 storage permissions.

• On the storage container: LIST
• On the storage directory: READ, WRITE and DELETE
• In the Access Control (IAM) settings for the ADLS Gen2 file system, assign the “Storage Blob Data Contributor” role to Replicate (AD App ID). It may take a few minutes for the role to take effect.

Setting Azure Data Lake Storage connection properties

To configure the connector, do the following:

1. In Connections, click Create connection.

2. Select the Azure Data Lake Storage target connector and then provide the following settings:

Data Target

Data gateway: Select the Data Movement gateway that will be used to test the connection to ADLS. This should be the same Data Movement gateway deployed to land data from the data source.

Connection properties

• Storage Account

  Name of the storage account.

• Container name

  Name of the container to use as cloud staging area.

• Azure Active Directory Tenant ID

  Tenant ID of the subscription in Azure Active Directory.

• Azure Application Registration Client ID

  Client ID of the application in Azure Active Directory.

• Azure Application Registration Secret

  Secret of the application in Azure Active Directory

Name

The display name for the connection.

Data type mapping

The following table shows the default mapping from Qlik Cloud data types to Azure Data Lake Storage data types.

Mapping from Qlik Cloud data types to Azure Data Lake Storage

Mapping from Qlik Cloud data types to Parquet

When Parquet is set as the file format, due to the limited number of data types supported by Parquet, the data type mappings will be as follows: