Early Access: The content on this website is provided for informational purposes only in connection with pre-General Availability Qlik Products.
All content is subject to change and is provided without warranty.
Skip to main content Skip to complementary content

Monitoring Qlik Open Lakehouse storage task

The storage task in Qlik Open Lakehouse projects differs from other projects as it runs continuously, rather than in batch mode. As a result, monitoring metrics are different.

General monitoring details

This section provides a summary of the overall task status:

  • Data task is updated to

    The timestamp until which all tables are fully synced. It represents the latest point in time where all source changes committed up to that moment are already available in the target tables.

    If no new source changes are detected, this value is set to the current time.

  • Number of datasets

    Total number of datasets included in this task.

  • Datasets with errors

    Total number of datasets in this task that encountered errors.

  • Lakehouse cluster

    The name and status of the cluster on which this task is running.

Full load monitoring details

You can view the following details for the data task in Full load status:

  • Queued: the number of tables currently queued.

  • Loading: the number of tables currently being loaded.

  • Completed: the number of tables completed.

  • Error: the number of tables in error.

You can view the following details for each table in the data task:

  • Name

    The name of the target table.

  • State

    Table state will be either: Queued, Loading, Completed, or Error.

  • Started

    The time that loading started.

  • Ended

    The time that loading ended.

  • Duration

    Duration of the load in format hh:mm:ss.

  • Records

    The number of records that were replicated during the load.

  • Message

    Displays error message if the load was not processed successfully.

Change data capture (CDC) monitoring details

You can view the following CDC details for the data task to monitor change processing in CDC status:

  • Incoming changes: the number of changes present at the source and waiting to be processed. You can view how many that are accumulated, and how many that are being applied.

  • Processed changes: the number of changes that have been processed and applied (in the last 24 hours).

  • Latency: the current latency of the data asset (hh:mm:ss). This duration represents the time from when the change is available in the source until the change is applied and available in the target or landing asset.

You can view the following details for each table in the data task:

  • Name

    The name of the target table in the landing asset.

  • State

    Table state will be either: Accumulating changes or Error.

  • Incoming changes

    The number of changes processed by the landing task which are waiting to be applied by the storage task.

  • Processed changes

    The total number of changes which was applied to the target since the dataset was last fully loaded (by the initial full load or by reload action).

  • Last processed

    The last source data time that was inserted to the target dataset, translated to local time.

  • Unoptimized changes

    Number of records that are queryable via the view, but not yet merged into optimized Iceberg partitions. High counts may affect performance until background optimizations finish.

    This reflects the number of records currently in the changes partition that have not yet been applied to the current or history partitions.

    If this number is consistently high, consider increasing cluster compute capacity to accelerate optimization.

  • Message

    Displays error message if changes to the table fail and are not processed.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!