Skip to main content

Select Partition Strategy for Each Topic

Topics and Partition Strategy

Kafka messages are distributed across partitions within a topic based on the Partition Key. Striim supports four combinations:

Single topic, one partition: All source data goes to one target topic, partition 0. Preferred when preserving the order of source DML operations.

Single topic, multiple partitions: All source data goes to one topic but is distributed across partitions. Order is preserved only within a single partition.

Multiple topics, single partition each: Wildcard or explicit mapping of source entities to topics; data is written to partition 0 of each topic.

Multiple topics, multiple partitions: Extension of the previous scenario --- source data is routed to multiple topics and further partitioned within each topic.

Configuring Partitioning

Partition Key

The Partition Key determines how events are distributed across partitions.

None: All messages are routed to partition 0. Use when ordering across all events is required.

Note

For OLTP CDC sources, if the source CDC Reader is configured with FilterTransactionBoundaries: false and all incoming events are mapped to a single topic with no partition key set, BEGIN and COMMIT control events from the source are also written to Kafka as messages.

Custom: Specify a field name or metadata reference as the partition key. Examples:

  • Field-based: PartitionKey = Custom, CustomPartitionKey = deptId

  • Metadata-based (OLTP CDC): CustomPartitionKey = @metadata(OperationName)

  • File-based: CustomPartitionKey = @metadata(directory)

UseMessageKey: The Message Key is also used as the Partition Key. This is useful when message ordering per entity is important.

Note

The specified partition key must be present in all incoming events (except control and DDL events when Persist Schema is OFF). If absent, the application will HALT.

Partitioning Method

Default --- Hash-Based Partitioning

Striim uses a hash-based algorithm by default:

Partition = hash(partition_key) % number_of_partitions

This guarantees that the same partition key always maps to the same partition, which is required for maintaining event ordering per entity.

Note

If the number of partitions in a topic is increased, partition assignments for existing keys will change. Always quiesce the application before altering the number of Kafka partitions. This also applies when changing the topic mapping configuration itself, not only the partition count.

Hash Collisions: If two different partition keys produce the same hash value, both keys are assigned to the same partition.

Custom Partitioning

The Kafka Writer supports custom partitioners via the Partitioner.class property in the Producer Configurations of the Kafka Connection Profile. Custom partitioners must implement the PartitionerIntf interface.

Important constraints:

  • One partitioning strategy per target --- the Kafka Writer applies the same partitioning logic to all topics within a single target.

  • For E1P, the partitioner must always return the same partition ID for a given partition key, even during retries.

Example RangePartitioner implementation:

package com.example.kafka; import com.striim.custom.partitioner.PartitionerIntf; import org.apache.kafka.common.Cluster; import java.util.List; import java.util.Map; public class RangePartitioner implements PartitionerIntf { private int rangeSize = 100; @Override public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { List<?> partitions = cluster.partitionsForTopic(topic); int numPartitions = partitions.size(); if (key == null) { return 0; } int keyInt; try { keyInt = Integer.parseInt(key.toString()); } catch (NumberFormatException e) { return 0; } return (keyInt / rangeSize) % numPartitions; } @Override public void close() {} }