Select Delivery Semantics
E1P --- Exactly-Once Processing
E1P is the default setting (E1P: true). In this mode:
Kafka transactions are used to ensure exactly-once delivery, avoiding duplicates even on retries.
A checkpointing topic is required. The writer creates it automatically if it does not exist (requires topic creation permission).
The transaction batch size is governed by the Commit Policy (see Kafka Writer Properties).
How the Commit Policy shapes transactions: if only EventCount is set, source transactions are buffered and combined until the event count is met — a single large source transaction can span multiple Kafka transactions. If only Interval is set, all source transactions received within that interval are combined into one Kafka transaction. If both are set, whichever condition is met first triggers the commit, and both counters reset.
Handling DDLs: on receiving a DDL event, the current buffer is rolled over and sent as its own transaction, even if the configured Commit Policy has not yet been met. DML events processed after the DDL use the newly evolved schema ID.
Required producer settings (set automatically): enable.idempotence=true and acks=all.
A1P --- At-Least-Once Processing
Set E1P: false to use at-least-once semantics. In this mode:
No checkpointing topic is required.
Duplicates may occur during retries --- acceptable if the downstream consumer is idempotent.
For A1P without duplicates on internal Kafka retries, configure the producer with: max.in.flight.requests.per.connection ≤ 5 and enable.idempotence=true.
Troubleshooting transactions
Handling large transactions: a large Commit Policy can buffer many events in memory and risk an out-of-memory error — increase available memory or use a smaller Commit Policy. If sending a transaction takes longer than the Kafka producer's transaction.timeout.ms (default 60000 ms), or the transaction isn't committed before it expires, the application may HALT. The producer's transaction.timeout.ms cannot exceed the broker's transaction.max.timeout.ms (default 900000 ms, 15 minutes).
Managing low throughput: increase ParallelThreads above 1, and tune batch.size via the Kafka Connection Profile's Producer Configuration. Failures during initProducerId typically indicate a transaction-coordinator initialization issue — verify broker health and ensure the transaction.state.log topic has a replication factor of at least 3. Note that Amazon MSK's broker defaults differ from Apache Kafka's: MSK defaults default.replication.factor and min.insync.replicas to 3 and 2 respectively (vs. 1 and 1 on Apache Kafka), while both default offsets.topic.replication.factor and transaction.state.log.replication.factor to 3; adjust manually on any cluster with fewer than 3 brokers.
Batching Support: batches for a given topic-partition are kept small to limit latency from batch-accumulation time and sent sequentially; batches across different topic-partitions are sent in parallel to increase throughput. The overall buffer size per target defaults to 32 MB (configurable via buffer.memory) and is the upper limit of batches accumulated across all topic-partitions before sending.
Parallel Threads distribution: when ParallelThreads is configured, each mapped topic is assigned to a writer instance using hash(topic_name) % parallel_threads, so all events for a given topic always go to the same instance. If several topic names happen to hash to the same instance, that instance becomes a hotspot; if this occurs, distribute the affected topics across separate Kafka Writer targets to balance the load.
Handling Retries
See also Retry and Recovery for a high-level overview. Two levels of retry occur:
Internal retry --- managed by the Kafka Producer Client. With E1P enabled, idempotent producers ensure no duplicates or out-of-order messages.
Writer-level retry --- Striim creates a new Kafka producer and retries all pending messages. This handles credential rotation and retriable exception codes. With A1P, duplicates can occur on connection glitches or connection information changes.