Viewing and downloading log files

You can review the logs for information about data tasks. You can also set the logging level and download the log files if needed. Logs are available as soon as the data task has been prepared.

Log files are available for the following task types:

• Landing

• Lake landing

• Replication

Viewing logs for data tasks based on Data Movement gateway

To view logs for data tasks that use Data Movement gateway, you need the following permissions:

• One of the following in the space where the data task resides:
  

  • Owner

  • Can operate

• One of the following in the space where the data gateway resides:

  • Can consume
  • Can edit
  • Can manage

To view the logs, do one of the following:

• In pipeline view, select View task logs from the menu in the bottom right of the data task.

• Open the data task and select View task logs from the menu at the far right of toolbar.

The log viewer opens.

If you are moving data from a SaaS application source, you can select Tasks logs or Source connection logs. Otherwise, only the tasks logs will be available.

You can scroll to the top and the bottom of the file with and .

Viewing logs for data tasks not using Data Movement gateway

To view the logs, do one of the following:

• In pipeline view, select View task logs from the menu in the bottom right of the data task.

• Open the data task and select View task logs from the menu at the far right of toolbar.

The log viewer opens.

The following logs are available:

• Task logs

  Information noteYou need Can operate permission in the space where the data task resides to view task logs.

  • Task logs

    The Task logs tab displays the log file currently selected in the drop-down list at the top of the tab. The number of logs in the drop-down list depends on whether or not the Show all task logs in drop-down list option is enabled. When this option is enabled, the drop-down list will contain all of the task logs, regardless of task completion time. When it is not enabled, the drop-down list will contain only the logs of tasks that completed up to theTask ended time.

  • Source connection logs (Available when using a SaaS application source only)

• Platform logs

  The Platform logs tab provides information for the platform components that facilitated the task execution.

  The platform log is not impacted by the task execution time. For example, the platform log might start at 12:35 and end at 13:30, even though the task started at 13:00 and ended at 13:15.

  Information noteYou need a Tenant Admin role to view platform logs.

  • Data Movement gateway logs (This is the internal Qlik-managed gateway)

  • Replication engine logs

  • Source connection logs (Available when using a SaaS application source only)

  • SaaS application logs

You can scroll to the top and the bottom of the file with and .

Setting logging options

You can set the level of logging for different operations of the replication under Logging options.

Storing trace and verbose logging in memory

When the logging level is set to "Trace" or "Verbose", you can store the logging information in memory until an error occurs. On detecting an error, the writing to the physical logs and continue to do so for a few minutes after the initial occurrence of the error.

If no error occurs before the allocated memory is used up, Qlik Talend Data Integration will empty the memory buffer and start afresh.

This option is useful for tasks that fail unpredictably and for no obvious reason. The problem with continually writing large amounts of information to the logs is twofold:

• Running in "Trace" or "Verbose" logging mode will quickly use up available disk space (unless the logging settings have been configured to prevent this).
• Continually writing large amounts of data to the logs will affect performance.

To use this option

1. Select the Store trace/verbose logging in memory, but if an error occurs, write to the logs check box at the top of the tab.
2. In the Allocate memory up to (MB) field, specify the amount of memory you want to allocate for storing logging information.

Setting logging levels

You can set the following levels:

• Error

  Show error messages.

• Warning

  Show warnings.

• Info

  Show informational messages.

• Debug

  Show additional information for troubleshooting purposes.

• Detailed debug

  Show detailed information for troubleshooting purposes.

The higher levels always include the messages from the lower levels. Therefore, if you select Error, only error messages are written to the log. However, if you select Info, informational messages, warnings, and error messages are included. In general, using the levels Debug and Detailed debug may generate large amounts of log data.

You can use Global to set the same level for all operations, or set the level individually for each operation.

• Source - full load

  Logs activity related to full load operations in the data source. This includes the SELECT statements executed against the source tables prior to full load.

• Source - CDC

  Logs activity related to CDC operations in the data source.

  Warning noteSetting this to Detailed debug level will generate very large amounts of data to the log.

• Source – data

  Detailed logging of data source activity related to full load and CDC operations.

• Target - full load

  Logs activity related to full load operations on the target.

• Target - CDC

  Logs activity related to CDC operations on the target.

• Target – upload

  Logs activity when files are transferred to the target.

• Extended CDC

  Detailed logging of CDC activity, such as synchronization and storage of transactions.

• Performance

  Logs latency values for source and target.

• Metadata

  Logs activity related to reading metadata, as well as metadata changes. Status of the replication task is also logged.

• Infrastructure

  Logs infrastructure information, file system operations, and task management.

• Transformation

  Logs information related to transformations that are performed.

Downloading files

You can download a diagnostic package, task logs, and a memory report to assist you when troubleshooting the replication task associated with the landing task. You can only download one item at a time.

1. In the log viewer, click Download to expand.

2. Select the item to download.

3. Click Download.

   The file will either be downloaded to your computer or you will be prompted to save it, depending on your browser settings.