Kafka Writer Operational Considerations
Monitoring and Metrics
The Kafka Writer publishes the following metrics for monitoring:
Metric Name | Description |
|---|---|
No. of Kafka Messages Sent | Total number of messages successfully published to Kafka across all topics and partitions since application start. |
Kafka Messages Rate | Number of messages published to Kafka per second. |
Kafka Write Rate | Average number of bytes sent per second to Kafka. |
Discarded Event Count | Total number of events discarded by the adapter. |
Buffered Messages Size (bytes) | Current memory usage for buffered messages or events in the adapter. |
Kafka Producer ID(s) | ID of the Kafka Producer instance used for publishing messages. |
Average Latency | Average time (ms) for Kafka request and acknowledgment per non-transactional buffer or transaction. |
Maximum Latency | Maximum time (ms) for Kafka request and acknowledgment per non-transactional buffer or transaction. |
Last Send Timestamp | Timestamp of the most recent successful Send operation to Kafka. |
No. of Retries | Total number of retry attempts made by the adapter due to connection or processing failures. |
Last Retry Attempt Timestamp | Timestamp of the most recent retry attempt. |
The following transactional metrics are visible only when E1P is enabled:
Metric Name | Description |
|---|---|
Successful Transactions | Total number of committed transactions executed. |
Average Messages per Transaction | Average number of events in each committed transaction. |
Average Transaction Size (bytes) | Average size of committed transactions in bytes. |
Max Transaction Size (bytes) | Maximum size in bytes of any committed transaction since application start. |
Average Transaction Accumulation Time (ms) | Average time spent accumulating events for a transaction. |
Commit Lag (ms) per Adapter Instance | Time taken between execution of a batch and its publication from each source, in milliseconds. |
Kafka Transactional ID(s) | ID of the Kafka Transactional Producer instance used for publishing messages. |
Per-topic and per-partition metrics are also available. For each topic, the adapter tracks: Mapped Source Entity, Total Kafka Messages Sent, Total DDLs Sent, Created By (if created during current run). For each partition within a topic: Latest Offset, Total No. of Messages, No. of DDLs, Avg Message Size, Last Activity Timestamp.
DDL Information metrics are also tracked per adapter instance: Schema Evolution (configured action: Auto or Manual), No. of DDLs Handled.
Troubleshooting
Known Limitations
Case Sensitivity
Case sensitivity is supported for topic names (see Topic Configuration) across all formatters, with the exception of the Avro formatter.
Handling CDC Events from MSJet
When using MSJet with KafkaWriter:
During Initial Load (IL), topics are created using a three-part name (database.schema.table).
During CDC processing, topics are created using a two-part name (schema.table).
This inconsistency may result in IL and CDC applications writing to different topics for the same table and can lead to schema registry conflicts when subject names are derived from topic names.
Workarounds:
Query + Tables combination --- use a three-part name in the query with a two-part name in the tables property. (Limitation: only one table can be configured per adapter).
Custom CQ --- apply a continuous query in the CDC application to enforce three-part naming in the metadata.
Prefixed wildcards --- use patterns such as %,<database>.% instead of full wildcards.
Pre-created topics --- disable AutoCreateTopic and configure explicitly created topics with direct mapping and SchemaRegistrySubjectName = UseTopicName.
Note
For scenarios involving multiple databases, create separate applications for each database rather than attaching multiple source adapters to a single target application.
Configuration Issues
Problem | Cause | Resolution |
|---|---|---|
Failed to parse Connection Retry Policy: Invalid key '<Key>' | The Connection Retry Policy contains an unrecognized key. | Use only valid keys: initialRetryDelay, retryDelayMultiplier, maxRetryDelay, maxAttempts, totalTimeout. Format: Key=Value;Key=Value. |
Connection Retry Policy '<key>' must be a valid time value | Value for a time-based parameter is not in valid format. | Use numeric milliseconds (5000) or time units (5s, 5m, 1h). No spaces between number and unit. |
Provide the Topics in this format: source,TopicName; | Topics property syntax is incorrect. | Single topic: use topic name only. Multiple: 'source,TopicName;source2,TopicName2'. Semicolons separate mappings; comma separates source from topic. |
Topic Key is not supported with single topic name | TopicKey set but Topics contains only a single topic name. | Remove TopicKey when using a single topic. Use topic mapping format if routing is needed. |
Topic Key is mandatory when using multiple topics | Topics has multiple mappings but TopicKey is not specified. | Set TopicKey to specify the routing field. Examples: @metadata(TableName) for CDC, field name for typed events. |
DataTopicConfig must be in valid JSON format | DataTopicConfig contains invalid JSON syntax. | Ensure valid JSON with double quotes for keys/strings. Example: {"PartitionCount":1,"ReplicationFactor":1}. |
CommitPolicy format invalid | CommitPolicy is not in EventCount=<n>;Interval=<duration> format. | Example: EventCount=1000;Interval=5m. Semicolon separates parameters; = separates key from value. |
Custom Partition Key must contain non-empty values | CustomPartitionKey has empty values between semicolons. | Use format: field1;field2;field3. Remove trailing/leading semicolons and empty values. |
Custom Message Key cannot be empty when MessageKey=Custom | MessageKey is Custom but CustomMessageKey is empty. | Configure CustomMessageKey with key=value pairs. Example: id=customerId;name=customerName. |
Timed out waiting for node assignment. Call: listTopics | Kafka AdminClient cannot connect to broker cluster. Broker addresses incorrect, network latency, firewall blocking, or brokers down. | Verify Broker Address is correct and reachable. Ensure firewall allows outbound connections to Kafka ports. Confirm brokers are running. |
Unexpected authentication error | Schema Registry URL is unreachable (wrong host/port, HTTP vs HTTPS mismatch, DNS/firewall/VPC). | Ensure the URL in the Connection Profile is valid and reachable. Verify the Schema Registry is running. |
Metadata/Userdata field not found: key <keyName> | A @metadata() or @userdata() key in TopicKey, PartitionKey, MessageKey, or MessageHeader does not exist in the incoming event. | Verify the field exists in the WAEvent metadata or userdata. Check spelling (case-sensitive). |
Topic <TopicName> does not exist and auto-create is disabled | The topic does not exist and AutoCreateTopic is false. Or user lacks DESCRIBE permission. | Set AutoCreateTopic=true, or manually create the topic. Ensure DESCRIBE ACL is granted on the topic. |
Cluster authorization failed | Kafka user lacks DESCRIBE_CONFIGS ACL required for auto topic creation with default settings. | Pre-create topics and disable AutoCreateTopic, or configure PartitionCount and ReplicationFactor explicitly in DataTopicConfig. |
Authorization failed | The authenticated Kafka user does not have all required ACLs. | Grant all required ACLs per the Required ACLs table. |
Retries and Error Recovery
See also Handling Retries for the underlying retry configuration. When E1P is on, exactly-once guarantees apply to both internal and external retries --- no duplicates will be produced.
With A1P:
Internal Kafka retries will not cause duplication or out-of-order messages if the producer is configured with max.in.flight.requests.per.connection ≤ 5 and enable.idempotence=true.
External (writer-level) retries create a new Kafka producer and retry all messages --- duplicates can occur on connection glitches or changes in connection information.
Handling Special Characters
Kafka topics created via Kafka Writer support dot (.), underscore (_), and hyphen (-).
Avro schemas registered via the Avro Formatter support: underscores and dots in namespace; underscores in Avro record names and field names.
In the legacy Kafka Writer (pre-5.4), special characters were converted to a mangled name format (_<Name of special character>_). The new Kafka Writer will HALT on unsupported special characters. Backward compatibility mode retains the mangled naming convention.
Avro naming limitation: The Avro Formatter currently uses Avro 1.7.7, which does not support special characters in Avro record names or namespaces. Only letters, numbers, and underscores are allowed, and the name must start with a letter or underscore.