Monitoring a data task

You can monitor the status and progress of a Storage, Transform, or Data mart task by clicking on Monitor.

• When the first load is running you can view progress in Full load status.

• While changes are processed you can view status and progress in Current batch of changes.

• When changes have been processed you can view status and progress in Last batch of changes.

In the lower half of Monitor you can view status and progress for each dataset.

You can also view detailed information on SQL statement level.

Viewing status and progress

You can view the following details for each dataset or change:

• State

  This shows the current state of this dataset or change.

  • Completed - the load or change has completed successfully.

  • Loading - the table or change is being processed.

  • Queued - the table or change is waiting to be processed

  • Error - there was an error when processing the table or change.

• Started

  The time that loading or change processing started.

• Ended

  The time that loading or change processing ended.

• Duration

  Duration of loading or change processing in format hh:mm:ss.

• Processed records

  The number of records processed in the load or change.

• Throughput (records/second)

  Throughput is not updated until the load is finished.

• Message

  Displays error message if the load or change was not processed successfully.

The datasets will be continuously updated with new data as the landing area is updated by the replication task. Each batch relates to records from a certain time span. You can see the time span of the most recent batch in Last batch of changes.

Data from all source transactions up to the time shown in Data asset is updated to is available for consumption from this data asset. This information is available for a data asset once all tables were loaded and the first set of changes applied. If you selected to generate live views, you can also view when live views are updated.

If there is a batch of changes before the initial load is completed, Data is updated to in the asset card will not be updated until the initial load is completed and the first batch of changes are applied. For example, assume that you are loading a data asset which contains an order dataset containing 1 million orders and an order details dataset containing 10 million order details. The datasets take 10 and 20 minutes to perform a full load, respectively. The order dataset is loaded first, followed by the order details dataset . While order dataset was loading, a new order was inserted. So when the order details are loaded, it may contain details of the new order, which does not yet exist in the order dataset . The order and order details datasets will only be in sync and fully updated to the same time after the first batch of changes is applied .

Viewing detailed information

You can view detailed information on SQL statement level.

1. Select Full load, Current batch of changes or Last batch of changes in the dropdown of the lower part of Monitor.

2. Select the datasets to monitor in detail.

3. Click Monitor details.

Monitor details is displayed, and you can view the commands that are executed for each step of the load or change process. You can click on a command to view the full SQL statements that were executed.

• Click Export to CSV to export a text file with full SQL statements for all listed commands.