Kinesis stream
Create an Amazon Kinesis stream to enable Qlik to capture and propagate instance state information from your lakehouse clusters.
A Kinesis stream is a real-time data streaming service that enables the collection, processing, and analysis of data as it is generated. It is designed to handle large volumes of streaming data with low latency, making it suitable for real-time monitoring, analytics, and messaging between distributed components.
In a Qlik Open Lakehouse deployment, a Kinesis stream is used to collect and share status and performance metrics, such as CPU usage, memory consumption, and health checks, for each instance within the lakehouse cluster.
Instances in the cluster do not communicate with each other directly. Instead, they poll the Kinesis stream to retrieve information about their peers. This mechanism allows the instances to coordinate distributed tasks, maintain awareness of each other's state, and enable scalable, fault-tolerant execution across the cluster.
Prerequisites
Ensure you have created the VPC, and subnet and Availability Zones for hosting your Qlik Open Lakehouse.
Creating a Kinesis stream
To create a Kinesis stream, do the following:
-
In the AWS console, go to Amazon Kinesis.
-
In Data Streams, click Create data stream and configure it:
-
Data stream name: Enter the name for your stream.
-
Capacity mode: Select the Provisioned option.
-
Provisioned shards: Ensure this value is set to 1.
-
Click Create data stream and make a note of the stream name.