Topic Configuration

Data can be routed to a single topic or distributed across multiple topics based on source table names, field values, or metadata. The Topics property accepts:

Topics can be pre-created in Kafka, or created automatically by the Kafka Writer when a mapped topic doesn't already exist. The AutoCreateTopic property is a boolean flag that controls this behavior — it applies to Data Topics only and has no effect on the Checkpoint Topic (see Checkpoint Topics below). AutoCreateTopic defaults to true.

When AutoCreateTopic is true, a Data Topic is created automatically the first time an event is received for a mapped topic that doesn't yet exist — this check happens once per mapped topic (across Striim application sessions), not on every event. Creating a topic requires CREATE and WRITE permission on the Kafka cluster.

If AutoCreateTopic is false and a mapped topic does not already exist, the application HALTs with a "Topic Not Found" exception.

If a topic with the resolved name already exists in Kafka, Kafka Writer uses it as-is and appends to it. Data Topics are never dropped by Striim — including ones Striim created itself; their lifecycle beyond creation is managed by the user.

Data Topic–specific configuration for auto-created topics (partition count, replication factor, etc.) is set via DataTopicConfig — see Topic Configuration below, not CheckpointTopicConfig, which applies only to the Checkpoint Topic.

bin/kafka-topics.sh \
  --create \
  --bootstrap-server <BROKER_HOST>:<PORT> \
  --replication-factor <REPLICATION_FACTOR> \  # Recommended: 3 or more (for fault tolerance)
  --partitions <NUM_PARTITIONS> \  # Use 1 if no parallel threads, or match the ParallelThreads count
  --topic <TOPIC_NAME> \
  --config cleanup.policy=compact

Data topics hold the actual Kafka messages produced from source events. Topic names are determined by the Topics property and the TopicKey field (for multi-topic scenarios).

Topics: EMP,employee-topic; src.DEPT,dept-topic; src.CUSTOMER,customer-topic
TopicKey: @metadata(TableName)

When E1P is enabled, Kafka Writer uses an additional checkpointing topic to store recovery information. If this topic does not exist, the writer creates it automatically (requires topic creation permissions on the Kafka cluster). The checkpointing topic configuration can be customized via the CheckpointTopicConfig property.

The Checkpoint Topic is created regardless of the AutoCreateTopic setting whenever E1P and recovery are both enabled. Kafka Writer reads the checkpoint topic using consumer isolation level read_committed.

Sample checkpoint-topic creation script (self-managed Kafka):

bin/kafka-topics.sh \
  --create \
  --bootstrap-server <BROKER_HOST>:<PORT> \
  --replication-factor <REPLICATION_FACTOR> \  # Recommended: 3 or more (for fault tolerance)
  --partitions <NUM_PARTITIONS> \  # Use 1 if no parallel threads, or match the ParallelThreads count
  --topic <TOPIC_NAME> \
  --config cleanup.policy=compact

By default the checkpoint topic is named <Fully Qualified Target Name>_CHECKPOINT; this can be overridden via the Checkpoint Topic property, which must be unique per Kafka Writer. The checkpoint topic is automatically dropped when the application is dropped.

If Kafka Writer lacks authorization to create or use the checkpoint topic, the application HALTs with an AdapterExternalException ("Client is authenticated but not authorized to perform requested operations"). The required permissions on the checkpoint topic are CREATE, DESCRIBE, DELETE, and DESCRIBE_CONFIGS.

When ParallelThreads is greater than one, a checkpoint topic created by Striim is created with a partition count equal to the thread count. If you supply an existing checkpoint topic, its partition count must match the thread count, or the application HALTs. When decreasing ParallelThreads, the extra checkpoint partitions are simply left unused. When increasing ParallelThreads, you must manually increase the checkpoint topic's partition count to match before restarting.

Events are discarded and not written to Kafka in the following scenarios:

The total count of discarded events can be monitored using the Discarded Event Count metric.