Schema evolution

Schema evolution allows users to easily detect structural changes to multiple data sources and then control how those changes will be applied to your project. Schema evolution can be used to detect all DDL changes that were made to the source database, although not all changes can be applied automatically (see "Supported data changes" below). Schema evolution can be performed using the web console or using the CLI as described below.

Required permission: Compose Designer or Admin.

Schema evolution requires certain options to be turned on in the Replicate task(s). For information about which options need to be enabled in Replicate for schema evolution, see Defining a Qlik Replicate task.

Supported data source changes

The following changes are supported:

• New columns - When this change is applied:
  • The new column will be added (as an attribute) to the end of the entity in the metadata
  • The column will be added to any mappings for that entity
  • The data warehouse will be adjusted
  • The storage task will be generated (but not run)
• New tables - When this change is applied:
  • The new table will be added (as an entity) to the metadata
  • A default mapping will be created
  • In the case of multiple storage tasks, the mapping will be added to the first task according to alphabetical order
  • The data warehouse will be adjusted
  • The storage task will be generated (but not run)

Schema evolution using the web console

1. Open the Schema Evolution window using one of the following methods:

   • Click the Schema Evolution button at the bottom of the METADATA panel.
   • Select Schema Evolution from the hamburger menu in the top right of the METADATA panel.
   • Open the Manage Metadata window and click the Schema Evolution toolbar button in the Logical Metadata tab.
2. Select which data sources to scan for changes. You can select either All data sources or Selected data sources. If you choose the latter, select which data sources to scan for changes.

3. Click Scan for Changes.

   During the scan, if a change is detected, the following message will be displayed:
   

   Found new DDL changes for data source <name>

   If no changes were detected, the following message will be displayed:

   No new DDL changes were found for data source <name>

   If unsupported changes were detected, one of the following messages will be displayed:

   DDL type <type> is not supported and will be skipped

   Column operation <operation> in data source <name> is not supported and will be skipped

4. When the scan completes, click Close.

   If changes were detected, the Schema Changes window opens showing the list of changes since the last scan (via the web console or the CLI).

5. Click Apply Changes Options if you want to apply the changes or Ignore Changes to close the window without applying the changes.

   When you click Apply Changes Options, the Apply Schema Changes window opens, showing the following apply options:

   • Apply changes to the metadata and the mappings - Select this option if you have other changes you would like to make before adjusting the data warehouse. You may even wish to undo some of the schema evolution changes as Compose does not allow you to choose which changes to apply.
   • Apply changes to the metadata and the mappings, and adjust the data warehouse - Select this option if you have other changes you would like to make before generating the tasks.
   • Apply changes to the metadata and the mappings, adjust the data warehouse, and generate relevant tasks - Select this option if you want to apply all the changes and generate associated tasks without making any prior changes.

6. Choose one of the options and click Apply Changes.

   The Applying Schema Changes window opens.

7. After the operations you selected in Step 5 have completed, click Close.

   Information note

   If, for whatever reason, Compose fails to add the changes to the metadata or the mappings, the next time you perform a scan, the changes will be detected again. However, if Compose succeeds to apply the changes but fails to adjust the data warehouse or generate the tasks, the changes will not be detected again. In such a case, you will need to manually adjust the data warehouse and/or generate the tasks after resolving the issue that prevented these operations from being performed automatically.

Schema evolution using the CLI

You can also use the Compose CLI to perform schema evolution. As opposed to the web console, the CLI does not output a list of changes, which makes it more suited to customers that wish to automate schema evolution through the use of scripts.

Command syntax:

ComposeCli.exe schema_evolution --project project_name [--data_sources data_source] --action apply_to_model_and_mappings|apply_to_model_mappings_adjust_storage|apply_to_model_mappings_adjust_storage_generate_tasks|ignore_changes

Parameters

Example

ComposeCli.exe schema_evolution --project MyProject --data_sources mysource1,mysource2 --action apply_to_model_mappings_adjust_storage