Kafka Writer operational considerations
Known Limitations
Case Sensitivity
Case sensitivity is supported for topic names (see Topic Configuration) across all formatters, with the exception of the Avro formatter.
Handling CDC Events from MSJet
When using MSJet with KafkaWriter:
During Initial Load (IL), topics are created using a three-part name (database.schema.table).
During CDC processing, topics are created using a two-part name (schema.table).
This inconsistency may result in IL and CDC applications writing to different topics for the same table and can lead to schema registry conflicts when subject names are derived from topic names.
Additional known limitations
FileReader with subdirectories: when using FileReader as the source with includeSubDirectories set to true and TopicKey set to @metadata(FileName), the FileName metadata value contains the full file path. Since the path includes directory separators, the resulting value does not comply with Kafka topic name rules. To avoid this, either disable includeSubDirectories on the source or use a different field for TopicKey.
Dropping NOT NULL or primary key constraints: when SchemaEvolution is set to Auto and the Avro Formatter's formatAs is set to Native or Table, dropping a NOT NULL or primary key constraint via DDL is not supported.
DDL column names with unsupported special characters: when SchemaEvolution is set to Auto and PersistSchema is On, the application HALTs if a DDL event contains column names with special characters that Avro doesn't support.
Workarounds:
Query + Tables combination --- use a three-part name in the query with a two-part name in the tables property. (Limitation: only one table can be configured per adapter).
Custom CQ --- apply a continuous query in the CDC application to enforce three-part naming in the metadata.
Prefixed wildcards --- use patterns such as %,<database>.% instead of full wildcards.
Pre-created topics --- disable AutoCreateTopic and configure explicitly created topics with direct mapping and SchemaRegistrySubjectName = UseTopicName.
Note
For scenarios involving multiple databases, create separate applications for each database rather than attaching multiple source adapters to a single target application.
Monitoring and Metrics
The Kafka Writer publishes the following metrics for monitoring:
Metric Name | Description |
|---|---|
No. of Kafka Messages Sent | Total number of messages successfully published to Kafka across all topics and partitions since application start. |
Kafka Messages Rate | Number of messages published to Kafka per second. |
Kafka Write Rate | Average number of bytes sent per second to Kafka. |
Discarded Event Count | Total number of events discarded by the adapter. |
Buffered Messages Size (bytes) | Current memory usage for buffered messages or events in the adapter. |
Kafka Producer ID(s) | ID of the Kafka Producer instance used for publishing messages. |
Average Latency | Average time (ms) for Kafka request and acknowledgment per non-transactional buffer or transaction. |
Maximum Latency | Maximum time (ms) for Kafka request and acknowledgment per non-transactional buffer or transaction. |
Last Send Timestamp | Timestamp of the most recent successful Send operation to Kafka. |
No. of Retries | Total number of retry attempts made by the adapter due to connection or processing failures. |
Last Retry Attempt Timestamp | Timestamp of the most recent retry attempt. |
The following transactional metrics are visible only when E1P is enabled:
Metric Name | Description |
|---|---|
Successful Transactions | Total number of committed transactions executed. |
Average Messages per Transaction | Average number of events in each committed transaction. |
Average Transaction Size (bytes) | Average size of committed transactions in bytes. |
Max Transaction Size (bytes) | Maximum size in bytes of any committed transaction since application start. |
Average Transaction Accumulation Time (ms) | Average time spent accumulating events for a transaction. |
Commit Lag (ms) per Adapter Instance | Time taken between execution of a batch and its publication from each source, in milliseconds. |
Kafka Transactional ID(s) | ID of the Kafka Transactional Producer instance used for publishing messages. |
Per-topic and per-partition metrics are also available. For each topic, the adapter tracks: Mapped Source Entity, Total Kafka Messages Sent, Total DDLs Sent, Created By (if created during current run). For each partition within a topic: Latest Offset, Total No. of Messages, No. of DDLs, Avg Message Size, Last Activity Timestamp.
A separate Last Transaction Information metric (JSON format, visible only when E1P is enabled) reports details of the most recently committed transaction per adapter instance: No. of Events in the Transaction, Transaction Size (bytes), Transaction Time (ms), Transaction Accumulation Time (ms), and Topic Partition Count, along with its own nested per-topic and per-partition breakdown scoped to that specific transaction.
DDL Information metrics are also tracked per adapter instance: Schema Evolution (configured action: Auto or Manual), No. of DDLs Handled.
Altering Application
The table below summarizes which Kafka Writer properties can be changed on a running application via ALTER, and what to expect afterward. Properties marked "Not recommended" or requiring a quiesce should only be changed with the application quiesced first, to avoid data loss or inconsistent distribution.
Property | Can be altered? | Expected changes after ALTER |
|---|---|---|
ConnectionProfileName | Yes | The application uses the newly attached connection profile. |
ConnectionRetryPolicy | Yes | The altered retry policy applies on the next connection failure. |
Topics | Not recommended (quiesce first) | Subsequent events are mapped based on the newly configured Topics. |
TopicKey | Not recommended (quiesce first) | Make sure Topics matches the new TopicKey before restarting; subsequent events are mapped accordingly. |
AutoCreateTopic | Yes | If changed true→false, topics must already exist. If changed false→true, new topics are auto-created. |
DataTopicConfig | Yes | Only affects topics created after the ALTER. |
PartitionKey / CustomPartitionKey | Not recommended (quiesce first) | Partition distribution changes for subsequent events. |
E1P | Yes (quiesce first) | Toggling requires Recovery to be on; switches between exactly-once and at-least-once semantics. |
CheckpointTopic | Not recommended (quiesce first) | Checkpointing moves to the newly configured topic; may cause data loss if not quiesced properly. |
CheckpointTopicConfig | Not recommended | Only takes effect if the checkpoint topic is (re)created after the ALTER. |
CommitPolicy | Yes | The new commit policy applies to subsequent events. |
ParallelThreads | Yes (quiesce first) | With E1P: decreasing is safe (extra checkpoint partitions go unused); increasing requires manually growing the checkpoint topic's partition count first, or the application HALTs. |
MessageHeader | Yes (quiesce first) | New headers apply to subsequent messages only; existing messages retain old headers. |
MessageKey / CustomMessageKey | Yes (quiesce first) | Affects message keys, and partition distribution if partitioning depends on the message key. |
Serializer | Not recommended (quiesce first) | Serialization format changes; consumers may need updating, or use a new topic. |
SchemaEvolution | Yes (quiesce first) | Changes DDL handling behavior for subsequent schema changes. |
PersistSchema | Yes (quiesce first) | If changed to false, DDL events stop being persisted to the data topic. |
Additionally: switching the Avro Formatter's formatAs from Table or Native to Default is supported while altering; switching from Default to Table or Native (or between Native and Table) is not supported and requires dropping and recreating the application with a different subject name.