Amazon S3

Amazon S3 can be used as:

• A cloud staging area when using Databricks (optional) or Amazon Redshift (required) as a data pipeline target. The cloud staging area is where data and changes are staged before being applied and stored.

• A target in a replication task.

  • For an explanation of how to replicate data to Amazon S3 with Standard, Premium, and Enterprise subscriptions, see Landing data in a data lake with a Standard, Premium, or Enterprise subscription.
  • For an explanation of how to replicate data to Amazon S3 with a Starter subscription, see Replicating data with a Qlik Talend Cloud Starter subscription.

Permissions required for landing data

• You must have an Amazon S3 bucket that is accessible from the Data Movement gateway machine.

  For information on signing up for Amazon S3, see http://aws.amazon.com/s3/.

• Bucket access credentials: Make a note of the bucket name, access key and secret access key - you will need to provide them in the Amazon S3 connector settings.

• Bucket access permissions: The following bucket access permissions are required:

  {
  	"Version": "2012-10-17",
  	"Statement": [
  	    {
  	     "Sid": "Stmt1497347821000",
  	     "Effect": "Allow",
  	     "Action": [
                  "s3:GetBucketLocation",
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::YOUR_BUCKET_NAME"
              ]
          },
          {
              "Sid": "Stmt1497344984000",
              "Effect": "Allow",
              "Action": [
                  "s3:PutObject",
                  "s3:GetObject",
                  "s3:DeleteObject"
              ],
              "Resource": [
                  "arn:aws:s3:::YOUR_BUCKET_NAME/target_path",
                  "arn:aws:s3:::YOUR_BUCKET_NAME/target_path/*"
              ]
          }
      ]
  }
  

Where YOUR_BUCKET_NAME is the name of your bucket and target_path is the intended location of the target files in your bucket.

Qlik Data Gateway - Data Movement prerequisites

Data Movement gateway connects to AWS using SSL. This requires an appropriate CA certificate to reside on the Data Movement gateway machine; otherwise, the connection will fail. The purpose of the CA certificate is to authenticate the ownership of the AWS server certificate.

Make sure that the required CA certificate exists in the following location on the Linux machine:

/etc/pki/tls/certs/ca-bundle.crt

If it does not exist, the simplest solution is to copy the certificates bundle from another Linux machine.

Setting Amazon S3 connection properties

To configure the connector, do the following:

1. In Connections, click Create connection.

2. Select the Amazon S3 target connector and then provide the following settings:

Data target

Data gateway: Select the Data Movement gateway that you want to use to access the target database.

Depending on your use case, this will either be the same Data Movement gateway deployed to land data from the data source, or a different one. For information about the possible Data Movement gateway deployment possibilities, see Common use cases.

Connection properties

• Access key: The access key for your Amazon S3 bucket.
• Secret key: The secret key for your Amazon S3 bucket.

• Bucket name: The name of your Amazon S3 bucket.

  Information note

  The default bucket region setting is auto-detect, which eliminates the need to set a specific region. However, due to security considerations, for some regions (for example, AWS GovCloud), you might need to explicitly set the region. In such a case, you can set the region code using the regionCode internal property.

  For a list of region codes, see the Region availability section in:https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html

  For instructions on setting internal properties, see below.

• Use AWS PrivateLink: Select this to connect to an Amazon VPC and then specify the VPC Endpoint URL (for example, https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com).

  Information noteThe Use AWS PrivateLink option is not supported when using an Amazon S3 bucket as the staging area for a Databricks target. For information on setting up connectivity to a Databricks target, see Databricks.

Internal properties

Internal properties are for special use cases and are therefore not exposed in the dialog. You should only use them if instructed by Qlik Support.

Use the and buttons to the right of the fields to add or remove properties as needed.

Name

The display name for the connection.

Data type mapping

The following table shows the default mapping from Qlik Cloud data types to Amazon S3 data types.

Mapping from Qlik Cloud data types to Amazon S3

Mapping from Qlik Cloud data types to Parquet

When Parquet is set as the file format, due to the limited number of data types supported by Parquet, the data type mappings will be as follows: