Monitoring Qlik Open Lakehouse storage task
The storage task in Qlik Open Lakehouse projects differs from other projects as it runs continuously, rather than in batch mode. As a result, monitoring metrics are different.
General monitoring details
This section provides a summary of the overall task status:
-
Data task is updated to
The timestamp until which all tables are fully synced. It represents the latest point in time where all source changes committed up to that moment are already available in the target tables.
If no new source changes are detected, this value is set to the current time.
-
Number of datasets
Total number of datasets included in this task.
-
Datasets with errors
Total number of datasets in this task that encountered errors.
-
Lakehouse cluster
The name and status of the cluster on which this task is running.
Full load monitoring details
You can view the following details for the data task in Full load status:
-
Queued: the number of tables currently queued.
-
Loading: the number of tables currently being loaded.
-
Completed: the number of tables completed.
-
Error: the number of tables in error.
You can view the following details for each table in the data task:
-
Name
The name of the target table.
-
State
Table state will be either: Queued, Loading, Completed, or Error.
-
Started
The time that loading started.
-
Ended
The time that loading ended.
-
Duration
Duration of the load in format hh:mm:ss.
-
Records
The number of records that were replicated during the load.
-
Message
Displays error message if the load was not processed successfully.
Change data capture (CDC) monitoring details
You can view the following CDC details for the data task to monitor change processing in CDC status:
-
Incoming changes: the number of changes present at the source and waiting to be processed. You can view how many that are accumulated, and how many that are being applied.
-
Processed changes: the number of changes that have been processed and applied (in the last 24 hours).
-
Latency: the current latency of the data asset (hh:mm:ss). This duration represents the time from when the change is available in the source until the change is applied and available in the target or landing asset.
You can view the following details for each table in the data task:
-
Name
The name of the target table in the landing asset.
-
State
Table state will be either: Accumulating changes or Error.
-
Incoming changes
The number of changes processed by the landing task which are waiting to be applied by the storage task.
-
Processed changes
The total number of changes which was applied to the target since the dataset was last fully loaded (by the initial full load or by reload action).
-
Last processed
The last source data time that was inserted to the target dataset, translated to local time.
-
Unoptimized changes
Number of records that are queryable via the view, but not yet merged into optimized Iceberg partitions. High counts may affect performance until background optimizations finish.
This reflects the number of records currently in the changes partition that have not yet been applied to the current or history partitions.
If this number is consistently high, consider increasing cluster compute capacity to accelerate optimization.
-
Message
Displays error message if changes to the table fail and are not processed.