Classes and Methods

To facilitate development (by the customer consumer developer(s)), Qlik provides a JAVA SDK packed as JAR for decoding its messages. The SDK is mainly built around the following classes/interfaces:

public interface MetadataStore { 

byte[] loadMetadata(String schemaId);

void saveMetadata(String schemaId, byte[] data);
}

This interface needs to be implemented by a customer class. Since the SDK cannot "guess" the behavior of the customer's code, the customer needs to supply a class implementing this interface.

The loadMetadata method locates data schemas (metadata) based on their ID.

The saveMetadata method stores the data schemas.

AttunityMessageDecoder

This class is used to decode messages read from Kafka:

public AttunityMessageDecoder(MetadataStore metadataStoreImplementor)

throws AttunityDecoderException 

A MetadataStore must be passed to the constructor, otherwise an exception will be thrown.

Methods

• public AttunityMessage decode(byte[] message) throws AttunityDecoderException – This is the main method to call where the byte array input parameter is the Kafka message bytes read from the Kafka topic. The method signature returns a common parent class named AttunityMessage but in fact it will return only child object instances of this parent which are the AttunityMetadataMessage class and the AttunityDataMessage class.

• public static String getVersion() - This method returns the Qlik Avro Decoder SDK version in the following format:

  major_version.minor_version.sp_version.build_id

  Example:

  1.0.0.3153

AttunityMetadataMessage

This class will be returned from the decoder SDK if the message is a metadata message.

Methods

The following methods return useful information:

• public String getSchemaId()

  Returns the 32 bytes ID of the data schema described by this metadata message

• public AttunityTableLineage getLineage()

  Returns table lineage information (class described later)

• public String getDataSchema()

  Returns the data schema required to decode data schemas conforming to the schema ID of this message

• public AttunityTableColumn[] getTableColumns()

  Returns information about the table columns included in this metadata message and subsequent data messages

• public Map<String, String> getHeaders()

  Inherited from its parent class (AttunityMessage), this method will return the headers included in the envelope message

• public GenericRecord getRawMessage()

  Inherited from its parent class (AttunityMessage), this method returns the raw decoded message as a Java Generic Record. This is often useful in the following cases:

  • When the metadata message needs to be serialized for future use.
  • When the existing SDK is older than the Qlik Replicate version writing the messages. In this case, some fields may not be returned by the methods described above and will only be accessible via the Generic Record interface.

AttunityDataMessage

This class will be returned from the decoder SDK if the message is a data message.

Methods

The following methods return useful information:

• public String getSchemaId()

  Returns the 32 bytes ID upon which the data message is based.

• public AttunityDataHeaders getDataHeaders()

  Returns data specific headers such as Transaction ID and Time. See also AttunityDataType below.

• public AttunityTableLineage getLineage()

  Returns table lineage information (class described later)

• public AttunityDataColumn[] getDataColumns()

  Returns the actual data columns as an array.

• public AttunityDataColumn[] getBeforeDataColumns()

  If the operation described in the data message is an update operation, this method returns the row columns before the change while the getDataColumns method returns the row columns after the change.

  As in the metadata class, the getHeaders and the getRawMessage methods are available as well as they are inherited from the same parent class (AttunityMessage). The usefulness of these methods is the same as in the metadata class.

AttunityTableLineage

This class is part of the AttunityMetadataMessage class.

Methods

• public String getServer()

  Returns the name of the Replicate Server performing the data replication.

• public String getTaskName()

  Returns the name of the Replicate task that is creating the messages.

• public String getSchemaName()

  Returns the source database schema name (e.g. dbo).

• public String getTableName()

  Returns the source table name.

• public long getTableVersion()

  Returns the table version number. The number starts with "0" when the task starts and is incremented each time a DDL change occurs. Note that if the task is restarted in Full Load mode, the number will be reset to "0".

• public Date getTimestamp()

  Returns the date/time when this message was created on the Replicate Server. This can be used to help track DDL changes in the source database.

AttunityTableColumn

This class is part of the AttunityMetadataMessage class.

Methods

• public String getName()

  Returns the column name.

• public AttunityAvroValueType getValueType()

  Returns a value type, describing the type of the column. This is not the original type on the source but rather the type forced by AVRO encoding supported types.

  The possible values are:

  BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTES, and STRING

• public AttunityDataType getDataType()

  Returns the AttunityDataType class (see description below).

• public int getOrdinal()

  Returns the ordinal position of the column.

• public int getPrimaryKeyPosition()

  Returns the position of table's primary key.

AttunityDataType

This class is part of the AttunityTableColumn class.

Methods

• public AttunityDataValueType getValueType()

  Returns the Qlik Replicate data type.

  Possible values are:

  DATE, TIME, DATETIME, BYTES, BLOB, REAL4, REAL8, INT1, INT2, INT4, INT8, UINT1, UINT2, UINT4, UINT8, NUMERIC, STRING, WSTRING, CLOB, NCLOB, and BOOLEAN

• public int getLength()

  Returns the length of the data column.

• public int getPrecision()

  Returns the precision of the data column.

• public int getScale()

  Returns the scale of the data column.

• public boolean isNullable()

  Returns a boolean value if the column is nullable.

AttunityDataHeaders

This class is part of the metadata class.

Methods

• public AttunityDataOperation getOperation()

  Returns the operation type performed on the row described by this data message. The possible operations are INSERT, UPDATE, DELETE, and REFRESH

  Note that REFRESH describes the Full Load replication mode while the other operation types describe Change Processing operations.

• public String getChangeSequence()

  Returns the operation’s source change sequence.

• public Date getTimestamp()

  Returns the date/time of the data operation.

• public String getStreamPosition()

  Returns the operation’s source stream position.

• public String getTransactionId()

  Returns the operation’s source transaction ID.

• public byte[] getChangeMask()

  Returns which data columns were changed in the source table.

  The change mask is a bitmask of data columns in little-endian order. The bit position in the change mask is based on the ordinal of the column in the metadata message of that table.

  This means that if there are 10 data columns, they occupy bits 0 to 9 in the bitmask.

  If UPDATE mask is 0B hexadecimal, which is 1011 binary – it means that the columns at ordinals 1, 2 and 4 were changed.

  The following describes the bit semantics:

  • For INSERT records, all non-null columns have the associated bits set.
  • For DELETE records, only primary-key (or unique index) columns have the associated bits set. This allows an applier to construct a DELETE statement without having to find the primary key fields from another source.
  • For UPDATE records, each column with a changed value will have the associated bit set.

• public byte[] getColumnMask()

  Returns which data columns are present in the message. Usually, this will include all of the table columns.

  Information note

  When replicating from an Oracle source without full supplemental logging, some columns might not be present in the data, since they could not be replicated.

  The column mask is a bitmask of data columns in little-endian order. The bit position in the column mask is based on the ordinal of the column in the metadata message for that table.

  This allows the applier to distinguish a null value that is the actual value of the column, from a null value that represents a column which could not be replicated from the source database.

AttunityDecoderException

This exception class can be thrown by the decode method in the AttunityMessageDecoder class.

Methods

The exception class has two public methods:

• public String getErrorMessage()

  A formatted error message describing the error.

• public AttunityDecoderExceptionCode getErrorCode()

  Returns the internal error code, which can be one of the following:

  • BAD_INPUT_MESSAGE

    The input message could not be parsed with the Qlik envelope schema.

  • BAD_MAGIC_VALUE

    The magic field in the Qlik envelope does not match the expected magic value (atMSG).

  • BAD_METADATA_MESSAGE

    The envelope was parsed successfully, but the metadata message could not be parsed using the embedded schema.

  • BAD_DATA_MESSAGE

    The envelope was parsed successfully, but the data message could not be parsed using the user supplied schema.

  • NO_METADATA_FOUND_IN_METADATA_STORE

    The envelope was parsed successfully and a data message was found, but calling the user supplied metadata store did not find a schema matching the data message ID.

  • UNRECOGNIZED_MESSAGE

    The envelope was parsed successfully, but the message type is not recognized by this module.

  • FAILED_TO_SAVE_INTO_METADATA_STORE

    The metadata message was created successfully, but saving bytes to the metadata store failed.

  • FAILED_TO_LOAD_FROM_METADATA_STORE

    The Envelope was parsed successfully, but loading bytes from the metadata store failed.