Overriding task settings for individual datasets when publishing to Kafka

When publishing to Kafka, you can override the data message publishing and message format task settings for individual datasets.

To do this:

1. Open the task. In the Datasets tab, click themenu to the right of the dataset and select Settings.

   The Dataset settings: <dataset name> dialog opens.

2. To override the data message publishing settings, select the Data message publishing tab and continue as described in Overriding data message publishing settings below.

3. To override the message format settings, select the Message format tab and continue as described in Overriding message format settings below.

Overriding data message publishing settings

For each setting, the current task setting is indicated by the field Task setting: <setting> (for example, Task setting: Separate topic for each).

For a description of these settings, see Data message publishing.

You can either retain the task setting or use the Expression Builder to specify a custom topic, message key, or partition key.

To use the Expression Builder:

1. Select the Custom topic, Custom message key, or Custom partition key field as appropriate, and click .

   Information noteCustom topics must already exist or the broker must be configured to create them.

   The Expression Builder opens.

2. Build an expression.

   You can use columns in the dataset, functions, and operators to build the expression.

   Click to move an item to the expression.

   You can also use all functions supported by the Kafka in the expression.

3. Click Extract parameters.

   You can now add a test value to Value to test for all parameters.

4. Click Test expression.

   You should now see the result of the expression, using the test values.

5. When you are ready, click OK.

6. The expression is now added to the field.

Building expressions

You need to build an expression that defines custom values. You can use:

• Columns

  All columns in the source dataset

• Functions

  String functions, numeric functions, null check functions, data and time functions, and logical functions.

  Information noteIf you use Azure Synapse Analytics as data platform, $RTRIM and $LTRIM will only trim space characters regardless of which character you specify to trim.

• Operators

  Mathematical and logical operators. Some operators are also available in the toolbar above the expression. This is also where you find the operator for string concatenation, ||.

Click to move an item to the expression.

You can also use all functions supported by Kafka in the expression.

Overriding message format settings

To override the task message attributes for this dataset, enable Use custom settings, and then configure the settings as described in Message attributes.