Skip to main content

Setting up Kafka Message Structure

Each incoming event is converted into a Kafka message (record) consisting of an optional header, a key, and a value (payload).

Message Header

See also Headers and Metadata for the conceptual overview. By default, messages are published without a Kafka header. You can define dynamic or static headers:

  • Dynamic header: reference fields from payload, metadata, or user data (only when the incoming stream contains typed events).

  • Static header: a fixed key-value pair applied to every message.

  • Multiple headers: use the UI widget or provide a semicolon-separated value in TQL.

Header format: key=value pairs. Example:

MessageHeader: 'TableName=@MetaData(TableName)'

Supported event types: JSONNodeEvent, WAEvent, AvroEvent, TypedEvent (all formatters).

Example: a static header MessageHeader: CName="StriimCluster" produces the Kafka header {"key":"CName","value":"StriimCluster"}. A dynamic header for an OLTP source, MessageHeader: Table=@metadata(TableName);Operation=@metadata(OperationName), produces two headers with the resolved table name and operation (e.g., INSERT).

Message Key

Three configurations are available for the Kafka message key:

None: Messages are published without a Kafka message key. This is the default. Applies to all event types.

Custom: Define a dynamic message key by referencing fields from payload, metadata, or user data, or add a static key. Applies to all event types.

PrimaryKey: The message key is automatically constructed from primary key column values of the source table (OLTP or DWH sources only). If no primary key columns are defined, all columns are used. Changes to PK definitions due to DDL are reflected in message keys when SchemaEvolution is set to Auto.

File-based sources and any source without a structured schema do not support primary-key-based message keys — selecting PrimaryKey for these sources causes the application to HALT. Also note: using PrimaryKey as the message key when the source table has no primary key defined increases the key size (since all columns are used) and can affect which messages are retained under log compaction — this configuration is not recommended.

Serialization

Message Keys can be serialized using one of the following formatters: JSON, Avro, DSV, XML.

When using the Avro Formatter, message keys are Avro records with key-value pair fields. The field type is based on the incoming event field's type for OLTP sources; for non-OLTP sources, the type is String. A separate schema is registered using the subject naming strategy with a -key suffix. If the message key is PrimaryKey and the event is a DDL with SchemaEvolution: Auto and PersistSchema: On, a separate schema is registered under the subject naming strategy with a -DDLKey suffix instead.

Mapping Partition Keys to Kafka Message Key

When PartitionKey is set to UseMessageKey, the Message Key value is also used to determine the target partition. This ensures that all events with the same message key always land in the same partition --- a common pattern for maintaining per-entity ordering.

Examples:

Scenario 1 (OLTP CDC):

MessageKey: Custom, CustomMessageKey: @metadata(TableName),@metadata(OperationName), PartitionKey: UseMessageKey

Scenario 2 (File-based):

MessageKey: Custom, CustomMessageKey: @metadata(directory);@metadata(FileName), PartitionKey: UseMessageKey

Scenario 3 (Primary Key):

If the source table has no primary key defined, distribution falls back to using all columns of the row as the key/partition basis.

MessageKey: PrimaryKey, PartitionKey: UseMessageKey --- distribution is based on primary key columns.

Message Payload

Each incoming event's contents (data, metadata, user data, and other fields depending on the Members configuration) are formatted and added to the Kafka message value. By default Striim uses the Striim Serializer. For Avro Formatters, this can be set to Confluent Serializer via the Serializer property.