Upgrading Qlik Data Gateway - Direct Access

New features and enhancements

• Process isolation is now turned on by default

  In previous versions, process isolation mode was turned off by default. From Direct Access gateway 1.6.9, it will be turned on by default, but only with a clean installation. If you are upgrading, your existing process isolation configuration will be preserved (off or on, depending on what it was before the upgrade).

  To learn more about process isolation, see Mitigating connector crashes during reload

• Removed configuration.properties file

  To prevent customers mistakenly modifying the wrong configuration.properties file, the configuration.properties template file has been removed from the Direct Access gateway installation folder. Now, there will be only one configuration.properties file, located in C:\ProgramData\Qlik\Gateway.

Resolved issues

Direct Access gateway 1.6.9 resolves the following issues with the chunk recovery feature:

• Chunks not deleted after reload

  Processed chunks would sometimes not be deleted from the disk after a reload.

• Chunks not deleted after a crash

  Processed chunks would sometimes not be deleted from the disk following a Direct Access gateway crash.

• High memory consumption

  High memory consumption would sometimes occur during reload of stored chunks.

For more information about the chunk recovery feature, see Storing corrupted or missing data chunks for future reloads

New features and enhancements

• Data chunks recovery mechanism

  This version introduces support for sending missing or corrupted data chunks. To control the size in memory of the reloaded data chunks, a new property has been added to the configuration.properties file: RELOAD_CACHE_MEMORY_MB

  To learn how to set this property, see Storing corrupted or missing data chunks for future reloads

• .NET 6.0.x no longer required

  From this version, running Direct Access gateway no longer requires .NET 6.0.x. Prior to installing Direct Access gateway, new customers should install .NET 4.8 (needed for the installation) as well as .NET 8.0 and ASP.NET Core Runtime (for working with Direct Access gateway) only.

  For more information about Direct Access gateway prerequisites, see System prerequisites

• Improved batch operations for dependent processes

  From this version, dependent processes will be checked in parallel instead of sequentially, thereby avoiding exceptions due to modification of the process list during runtime.

  For more information on process isolation, see Mitigating connector crashes during reload

• Improved troubleshooting

  More context (for example such as "process ID" and "commit size") was added to the resource usage warnings in the logs.

Resolved issues

Direct Access gateway 1.6.8 resolves the following issues:

• Misleading error message

  Following a restart to the gateway (either manually or due to a crash), requests might be sent to the connector agent (ConnectorAgent.exe) before it has been initialized. In such a case, the error message would be misleading. Now, the error message will indicate that the gateway is disconnected.

• Reload failures due to disconnection

  Direct Access gateway disconnections after the initial load would sometimes cause subsequent reload to fail, with the following error:

  Requested endpoint could not be provisioned due to failure to acquire a load slot: Command getReloadSlot error for reload

  Reload failures due to pod shutdown

  The Analytics engine would sometimes cache connections from earlier sessions and send a request to a pod that had been requested to shutdown. This would cause reloads to fail with the following error:

  Connector error: data loading error (DirectAccess-4008)

• Reload stuck after restarting the Direct Access gateway service

  In certain scenarios, the connector-agent would not attempt to reconnect following a disconnection during the reload, resulting in the following error:

  Reload web socket was aborted due to a failure (DirectAccess-1103)

• Amazon Redshift connector: SELECT table would populate duplicate columns

  When tables existed in an external schema that was referenced from the database created in Datashare, duplicate columns would be shown in the DLE/ Data Manager preview.

New features and enhancements

• Support for automatic port allocation when working with process isolation

  Previously, it was not possible to set the [ODBC|SAPBW|SAPSQL|SAPPACKAGE]_MAX_PROCESS_COUNT value higher than 10 when using the default port number and range settings. From this version, ports will be allocated automatically, allowing the number of concurrent reload processes to be greater than 10.

• Enhanced metadata paging when working with the SQP SQL (via Direct Access gateway) connector

  With this enhancement, when 10,000 tables or more are retrieved, the tables will be retrieved in batches of 10,000 to prevent timeouts.

Resolved issues

Direct Access gateway 1.6.7 resolves the following issues:

• Expired reloads not released when using process isolation

  When using process isolation, connector processes would not be released for expired reloads.

• Amazon Redshift sessions continue running despite canceling the app

  Amazon Redshift sessions would continue running until the query finished, even if the running QCS app was canceled.

• Resolved security vulnerabilities

  • Direct Access gateway connectors would bind all services to 0.0.0.0, allows connections from every IP address on the local machine, potentially exposing the server to every possible network.
  • Direct Access gateway would attempt to load MPR.dll from a non-existent, dynamically generated directory within the user's temporary files path. Due to the absence of MPR.dll in the expected location, an attacker could place a malicious DLL in this path, which the installer could subsequently load and execute.

• Reload stuck after restarting the Direct Access gateway service

  When using non-select queries and a Stored Procedure, the reload would hang for several hours after restarting the Direct Access gateway service.

• Data preview and reload issues

  Data preview and reload issues would sometimes be encountered when using the ODBC (via Direct Access gateway) connector to access Microsoft Fabric.

• Error when adding an ORDER BY clause

  Adding an ORDER BY clause would cause an error when using the ODBC (via Direct Access gateway) connector and SQL Anywhere 17.

New features and enhancements

Direct Access gateway 1.6.6 introduces the following features and enhancements:

• Support for accessing ServiceNow via Direct Access gateway

  This version introduces support for the ServiceNow (via Direct Access gateway) connector, allowing customers to securely access firewalled ServiceNow data sources over a strictly outbound, encrypted, and mutually authenticated connection.

• Support for new authentication methods when using the following Direct Access gateway connectors:

  • Azure Synapse Analytics: Azure OAuth and Azure Active Directory
  • Azure SQL Database: Azure OAuth and Azure Active Directory
  • Snowflake: OAuth
  • Databricks: Azure OAuth
  • MongoDB: SCRAM-SHA-256 and LDAP

• Enhanced metadata paging when working with ODBC connectors

  With this enhancement, when 10,000 tables or more are retrieved, the tables will be retrieved in batches of 10,000 to prevent timeouts.

• .NET 8.x requirement

  From this version, customers will be required to install both .NET 6 and .NET 8.x in order to work with Direct Access gateway. During installation or upgrade, setup will install them automatically if they are not already installed.

• Improved supportability

  To aid troubleshooting, additional logging has been added that measures the duration of Direct Access gateway requests.

Resolved issues

Direct Access gateway 1.6.6 resolves the following issues:

• If the connection between Direct Access gateway and Qlik Cloud was idle for 30 seconds or more, the reload operation would sometimes fail with the following error: Invalid object name '#<TempTableName>' (indicating that the Direct Access gateway temp tables are unavailable).

  To remedy this, a new ODBC_RELOAD_SESSION_LIFE parameter was added to the C:\ProgramData\Qlik\Gateway\<GATEWAY-INSTALL-DIR>\configuration.properties file. The new parameter sets the reload session duration for ODBC connectors and has a default value of 20 seconds.

• Canceling a reload would sometimes cause an ODBC process creation and termination loop.
• The connection between Direct Access gateway and Qlik Cloud Services would sometimes end abruptly, without apparent cause.

• The Direct Access gateway connection retry interval would grow continually but never reset itself. Restarting the Direct Access gateway service would be the only way to reset the retry interval.

• When using incorrect syntax to call a Stored Procedure from Oracle, the reload would operation would hang for several hours instead of terminating immediately with an informative error.

New features and enhancements

Direct Access gateway 1.6.5 introduces the following features and enhancements:

• Support for the SAP ODP connector

  This version introduces the ability to access a SAP ODP source via Direct Access gateway. To facilitate this, a new Data gateway field has been added to the SAP ODP connector. The new field allows customers to choose which gateway to use to access their SAP ODP source.

  See also: SAP ODP Connector

• Extended logging capabilities

  This version introduces a new logger that Qlik Support can use to troubleshoot issues with specific gateway components.

• Newly supported region

  Japan is now supported as a Qlik Cloud tenant region.

• Performance enhancement

  Improved memory allocations for encrypted sessions.

Resolved issues

Direct Access gateway 1.6.5 resolves the following issues:

• Security vulnerability

  Secrets would be written to the DCAAS log when working with the ODBC connector.

• Process isolation performance

  Processes would take a long time to start under heavy load.

• Incorrect installation path

  Installing Direct Access gateway would always set Qlik as the root folder, even if the user chose a different folder in the setup wizard.

New features and enhancements

• Support for viewing and downloading log files from the Administration activity center

  This version introduces the ability to view Direct Access gateway log files in the Administration activity center and share them with Qlik Support if needed. Under normal circumstances, there is no reason to view the log files, but should you experience repeated failures or other issues, the log files will often help pinpoint the cause.

  See also: Viewing and downloading log files

• Support for installing Direct Access gateway silently

  Customers can now install Direct Access gateway silently, which is useful if you need to install Direct Access gateway on several machines throughout your organization.

  See also: Silently installing Direct Access gateway

• Improved session continuity

  When a reload session expired, a DirectAccess-1012 error would sometimes occur. Now, Direct Access gateway will try to keep the reload session alive by pinging Qlik Cloud more frequently.

Resolved issues

Direct Access gateway 1.6.4 resolves the following issues:

• Incorrect parameter name in the configuration.properties file

  When performing a new installation of Direct Access gateway, the EXTEND_FIRST_REQUEST_TIMEOUT parameter name in the C:\ProgramData\Qlik\Gateway\configuration.properties file would incorrectly be called FIRST_REQUEST_TIME_EXTENSION.

• Reload pod availability

  When multiple reloads shared the same reload ID, the number of reload pods in QCS would reach the maximum.

Direct Access gateway 1.6.3 includes the following enhancements and bug fixes:

• Data loss prevention

  In previous versions, to prevent potential data loss, Direct Access gateway did not automatically reconnect to the data source during a reload. From this version, Direct Access gateway will now automatically reconnect to the data source as needed. To prevent data loss, the completed reload summary will be verified before reconnecting. If data loss is detected, the reload will fail.

• Timeout for reload sessions

  In rare scenarios, the reload session would stay alive indefinitely. To prevent such occurrences, this version introduces a three-hour timeout for reload sessions.

• Timeout extension for load requests

  Reloads would sometimes fail with the following errors:

  Connector error: The session does not exist (DirectAccess-1509)

  Connector error: The stream session expired for the load data request (DirectAccess-1012)

  To mitigate this issue, an EXTEND_FIRST_REQUEST_TIMEOUT parameter (for extending the load request timeout) was added to the C:\ProgramData\Qlik\Gateway\configuration.properties file. The unit of measurement for the parameter value is minutes. Customers who frequently encounter these errors should try setting the value to 5-10 minutes.

  For detailed instructions, see Troubleshooting Direct Access gateway

• Support for detailed connection error messages

  Added support for viewing detailed connection error messages when using ODBC-based via Direct Access gateway connectors. Customers can now set the ShowErrorDetailMessage parameter in the Advanced connector settings to "true" to see the full error returned by the ODBC driver. The detailed error message will be show in the Data load editor.

  Information noteThe error message is provided in its entirety by the data source you are using. Qlik does not append any additional information to the error message or edit it in any way.

• Handling multiple reloads with the same reload ID

  From this version, the engine session ID will be used to distinguish between multiple reloads with the same reload ID. This will prevent potential conflicts, resulting in reloads failing with the following error:

  Connector error: data loading error (DirectAccess-4008

Direct Access gateway 1.6.2 resolves an issue that would sometimes cause the Connector Agent to crash when working in process isolation mode.

New features and improvements

Direct Access gateway 1.6.1 introduces several important features and enhancements, improving overall stability and performance.

• Support for spreading reloads across multiple connector processes

  Connector process isolation has been added to Direct Access gateway. This is an opt-in feature, meaning that it is off by default. Process isolation allows administrators with direct access to the gateway server operating system to control how many processes are available for reloads per connector type. Spreading reloads across multiple connector processes increases the level of isolation by reducing the number of reloads running concurrently on each connector process. This in turn can improve overall Direct Access gateway stability in cases where connector crashes have occurred.

  See also: Mitigating connector crashes during reload

• Added the ability to disable chunking when working with the SAP BW (via Direct Access gateway) connector

  This version introduces the BEX_ONE_CHUNK application parameter for the SAP BW (via Direct Access gateway) connector. When accessing SAP BW directly from Qlik Cloud (that is, not via Direct access gateway), to optimize performance, the result set is divided into chunks in the SAP backend and read as separate threads. However, when accessing SAP BW via Direct Access gateway, dividing the result set into chunks has no real advantage. On the contrary, it consumes more of the shared memory in SAP as it requires an additional temp table to be created for the duration of the data transfer. It's possible (though usually not required) to start chunking by setting BEX_ONE_CHUNK:false in the appsettings.json file, which is located in the SAP connector binaries folder.

• Diagnostics improvements for SAP BW (via Direct Access gateway) and SAP SQL (via Direct Access gateway) connectors

  For SAP BW (via Direct Access gateway) and SAP SQL (via Direct Access gateway) connector logs, a _port suffix was added before the log file name extension, and a ProcessID=### message was added to the connector's log files in order to distinguish between events from new instances of the connector listening on the same port.

• Reconnection improvements

  The reconnection mechanism during reload has been improved so that Direct Access gateway no longer needs to connect to the pod that started the reload (as was previously the case). Now, instead of repeatedly trying to connect to the pod that started the reload (which often resulted in reload failures), it will connect to the first available pod.

Resolved issues

Direct Access gateway 1.6.0 resolves the following issues:

• ODBC via Direct Access gateway connector log:

  When reloads were performed from different data sources, processes from one data source would erroneously appear in the logging of the other data source.

• Missing file in setup

  The configuration.properties file would not be copied during the Direct Access gateway installation.

• Reload error during Qlik Cloud resource scaling

  Qlik Cloud resource scaling would sometimes result in a DirectAccess-1006 error. An enhancement was made to improve handling of certain Qlik Cloud resource scaling scenarios that occur during application reloads performed via Direct Access gateway.

• Snowflake connection failure

  Connecting to Snowflake via the ODBC (via Direct Access gateway) connector would fail.

• Debugging application script failure in DLE

  Debugging application scripts in the Data Load Editor (DLE) with the Limit Results option enabled would sometimes fail with the following error:

  Connector error: Invalid load request (DirectAccess-1006).

Resolved issues:

• ODBC (via Direct Access gateway) connector: The connector would sometimes crash when canceling reloads of large tables. An improvement was made to help prevent this.

• Microsoft SQL Server (via Direct Access gateway) connector: When attempting to establish a connection to a SQL Server hosted on older Windows Server versions, the connection would fail due to a TLS compatibility issue and the following error would be returned:

  Please check the values for Username‚ Password‚ Host and other properties. Description: Client unable to establish connection - ERROR [08001] [Qlik][SqlServer] Problem establishing connection to the server.

  The SQL Server driver has been updated to correct the problem.

• SAP SQL (via Direct Access gateway) connector: When attempting to preview a table, the following error would sometimes occur:

  Exception of Type System.OutOfMemoryException

• SAP SQL (via Direct Access gateway) connector: Attempting to preview a table would sometimes time out after a few minutes with the following error:

  Request could not be completed.

• SAP SQL (via Direct Access gateway) and SAP BW (via Direct Access gateway) connectors: Added diagnostic information to the logs to assist in troubleshooting multiple concurrent requests.

New features and enhancements:

• Provides support for the new ODBC (via Direct Access gateway) connector.

• The Version column in the Administration activity center will now display the Direct Access gateway version instead of the RepAgent (one of its sub-components) version.

Resolved issues:

• Upgrading Direct Access gateway would reset advanced ODBC Connector logging settings (such as log file retention and maximum size) to default.
• In rare cases, the ODBC connector would crash and could not be restarted by the Direct Access gateway agent.
• When a connection failed or the SQL query was not valid, a generic "Offset and length were out of bounds..." error would be returned instead of the actual error cause.
• In rare scenarios, hub reloads would fail with a "DirectAccess-4008" error for apps larger than 3 GB that had extended delays or processing after the initial gateway query. Additional logging has been added to assist in identifying any related issues.

Resolved issues:

• Changing the ODBC_LOG_LEVEL value in the C:\ProgramData\Qlik\Gateway\configuration.properties file would have no effect.

• Reload would fail with the following error if an active Direct Access gateway connection was not used for several minutes:

  Connector error: Invalid load request (DirectAccess-1006)

  The issue was resolved by not timing out if the session is still alive.

• Loading tables with numerous columns from MySQL and PostgreSQL would sometimes fail with the following error:

  System.InvalidOperationException: Failed to HPACK encode the headers

• When users did not have privileges on a Managed Space containing the Direct Access gateway, the following unclear error would be shown:

  Requested endpoint could not be provisioned due to failure to acquire a load slot: Object reference not set to an instance of an object.

  The error message was improved with a more accurate description of the issue.

• Multiple reload attempts would sometimes fail due to excessive memory consumption.

Features and enhancements:

Direct Access gateway checks its health periodically. Starting from this version, the health check will restart the Direct Access gateway service if it is unresponsive for five minutes.

Resolved issues:

• When starting a reload with a query that returns metadata immediately but only returns the first data chunk several minutes later, the following error would sometimes occur:

  Connector error: data loading error (DirectAccess-4008)

• In certain scenarios, Direct Access gateway would not reconnect to Qlik Cloud when a reload was interrupted.

• In rare situations, the following error would be encountered during reloads:

  No connection could be made because the target machine actively refused it.

• When connecting to SQL Server, special characters in passwords would not be encoded correctly, resulting in the following error:

  Command test connection returned non-success: Error message: Please check the values for Username‚ Password‚ Host and other properties. Description: Access credentials are not valid for this connection.

• The ODBC connector would sometimes crash when aborting load activity from the user interface while the query was returning data.

• In rare scenarios related to heavy loads, reloads would sometimes fail with the following error:

  Connector error: Unknown reason:. Index was outside the bounds of the array. (DirectAccess-1500)

Updated the CA certificates needed for authenticating the Qlik Cloud tenant.

This version resolves the following issue:

Qlik Data Gateway - Direct Access 1.3.1 is prone to excessive CPU utilization in certain scenarios. Version 1.3.2 resolves this issue and contains no other changes. To prevent CPU-related failures, we encourage all customers to upgrade to version 1.3.2 immediately.

This version includes the CA bundle for Qlik Cloud Government and resolves several issues.

CA bundle file for Qlik Cloud Government

The CA bundle authenticates the identity of the Qlik Cloud tenant, thereby ensuring a trusted connection.

In previous Direct Access gateway versions, the CA bundle file for Qlik Cloud Government customers was provided by Qlik as needed. From this version, the CA bundle file is included in the installation.

In a default setup, the CA bundle file is installed to the following location: C:\Program Files\Qlik\ConnectorAgent\caBundle]qcg_ca_bundle.pem

Resolved issues

• A security vulnerability was identified whereby the HTTP client used for proxy connectivity would follow a redirect.

• The Direct Access gateway agent would sometimes crash with the following error:

  Unable to write data to the transport connection: An established connection was aborted by the software in your host machine

• An exception would sometimes occur during log initialization, which would cause the Direct Access gateway to hang on restart.
• Reload session handling was improved to prevent unnecessary timeouts.
• The Direct Access gateway Version column in the Administration activity center would indicate the wrong version.
• The agent would continue to send data to the gateway after the user stopped the task or the quota limit was reached.
• End-to-end encryption for Data Gateway would not be enabled by default and was controlled by runtime flags.

This version resolves the following issues:

• Version 1.2 of Qlik Data Gateway – Direct Access might be prevented from installing due to an issue with its digital signature.
• Configuration files would be overwritten during upgrade, resulting in changes being lost

Features and enhancements:

• Added support for SAP BW and SAP SQL sources. For information on setting up connectivity to these sources, see SAP NetWeaver.

Resolved issues:

• Improved stability and recoverability of the Qlik Data Gateway - Direct Access service, eliminating the need to manually restart the service
• Support for overwriting the default Direct Access gateway installation and configuration parameters

Features and enhancements:

• Added Windows Server 2022 support

Resolved issues:

• Resolved a potential stability issue related to choosing a custom path during setup

1.1.0

October 18, 2022

Features and enhancements:

• Support for Oracle Wallets
• Support for Google BigQuery
• Improved error messages

Resolved issues:

• Improved system recoverability after driver crashes
• The Qlik Data Gateway - Direct Access service would sometimes stop unexpectedly