Operational Considerations
For certificate roatation, see Certificate rotation for Kafka Writer.
Retries
Note the following regarding retry behavior:
When E1P is turned on, it guarantees no duplicates for both Internal and External retries
With A1P Internal Kafka retries will not cause any duplication or messages out of order if Producer is configured with following properties:
max.in.flight.requests.per.connection <= 5 enable.idempotence=true
When idempotence is set to true a producer id and sequence number will be assigned to each kafka message. This helps Kafka Producer clients to deduplicate messages. During external retries (Writer-level) a new Kafka producer will be created and all the messages will be retried. So we can have duplicates when there is a retry due to connection glitches or changes in connection information.
Refer to this link to understand how idempotence works in Kafka.
Monitoring
The writer supports detailed monitoring metrics as follows:
Metric name | Readable name [in UI and MON output] | Sub-Metric readable name [in UI and MON output] | Sub-Metric readable name [in UI and MON output] | Sub-Metric readable name [in UI and MON output] | Applicable/ Visible when | Measure | Scope | Updated | Description |
|---|---|---|---|---|---|---|---|---|---|
TOTAL_KAFKA_MESSAGES_PUBLISHED | No of Kafka messages Sent | Always | Count of messages successfully published to Kafka | Since the application started | After each message sent / transaction execution | Total number of messages successfully published to Kafka across all topics and partitions since the start of the application. | |||
KAFKA_MESSAGES_RATE | Kafka Messages Rate | Always | Rate of messages published per second | Since the application started | After each message sent / transaction execution | Number of messages published to Kafka per second | |||
KAFKA_BYTES_RATE | Kafka Write Rate | Always | Average bytes sent per second | Since the application started | After each message sent / transaction execution | Average number of bytes sent per second to Kafka | |||
DISCARDED_EVENT_COUNT | Discarded Event Count | Always | Count of events discarded | Since the application started | When events are discarded | Number of events that were discarded | |||
WRITER_METRICS | Writer Metrics | Total No. of Transactions | When E1P is enabled | Count of transactions Committed | Since the application started | After each transaction execution | Total number of transactions executed when transactional support is enabled. | ||
Avg Messages per Transaction | When E1P is enabled | Average count of messages per transaction | Since the application started | After each transaction execution | Average number of events in each committed transaction, calculated as a rolling average. | ||||
Avg Transaction Size (in bytes) | When E1P is enabled | Average size in bytes | Since the application started | After each transaction execution | Average size in bytes of committed transactions, providing insight into transaction payload sizes. | ||||
Max Transaction Size (in bytes) | When E1P is enabled | Maximum size observed in bytes | Since the application started | After each transaction execution | Maximum size in bytes of any committed transaction since the start of the process. | ||||
Avg Transaction Accumulation Time (in ms) | When E1P is enabled | Average time in milliseconds | Since the application started | After each transaction execution | Average time spent accumulating events for a transaction. | ||||
Buffered Messages Size (in bytes) | Always | Current memory usage in bytes | Since the application started | Continuously updated - after receiving every event and after each message/transaction execution | Current memory usage for buffered messages/events in the adapter. | ||||
Commit Lag (in ms) | When E1P is enabled | Time difference in milliseconds | Since the application started | After each transaction execution | Time taken between execution of a batch to when it was published from each source in ms | ||||
Kafka Producer ID | Always | Producer identifier | Since the application started | At initialization | ID of the Kafka Producer instance used for publishing messages. | ||||
Kafka Transactional ID | When E1P is enabled | Identifier of the transactional producer | Since the application started | At initialization | ID of the Kafka’s Transactional Producer instance used for publishing messages. | ||||
Avg Transaction Time (in ms) | When E1P is enabled | Average time in milliseconds | Since the application started | After each transaction execution | Average time taken to complete Kafka request and acknowledgment for a transaction. | ||||
Avg Send Time (in ms) | When E1P is disabled | Average time in milliseconds | Since the application started | After each non transactional buffer execution | Average time taken to complete Kafka request and acknowledgment for a kafka message. | ||||
Max Transaction Time (in ms) | When E1P is enabled | Maximum time in milliseconds | Since the application started | After each transaction execution | Maximum time observed for Kafka I/O operations during any transaction. | ||||
Last Commit Timestamp | When E1P is enabled | Timestamp of last commit | Most recent commit | After each transaction execution | Timestamp of the most recent successful commit operation. | ||||
Last Send Timestamp | When E1P is disabled | Timestamp of last Send call | Most recent Message Ack | After each non transactional message ack | Timestamp of the most recent successful Kafka write operation. | ||||
No. of Retries | Always | Count of retry attempts | Since the application started | When retry attempts are made | Total number of retry attempts made by the adapter due to connection or processing failures. | ||||
Last Retry Attempt Timestamp | Always | Timestamp of the last retry attempt | Most recent retry | When retry attempts are made | The timestamp when the last retry attempt was made by the adapter. | ||||
CDDL Action | Always | Action type | Since the application started | At initialization | Action configured for DDL: Process, Halt, or Ignore. | ||||
No. of DDLs Handled | Always | Count of DDLs processed | Since the application started | When DDL events are processed | Number of DDLs processed according to the configured CDDL action. | ||||
LAST_TRANSACTION_INFO | Last Transaction Information | No. of Events in the Transaction | When E1P is enabled | Count of events | Most recent transaction | After each transaction execution | Number of events in the most recent committed transaction. | ||
Transaction size (in bytes) | When E1P is enabled | Size in bytes | Most recent transaction | After each transaction execution | Size of the most recent transaction in bytes. | ||||
Transaction Time (in ms) | When E1P is enabled | Time in milliseconds | Most recent transaction | After each transaction execution | Time taken to complete the last Kafka I/O operation for the transaction. | ||||
Transaction Accumulation Time (in ms) | When E1P is enabled | Time in milliseconds | Most recent transaction | After each transaction execution | Time spent accumulating events before forming the batch for the last transaction. | ||||
Topic Partition Count | When E1P is enabled | Count of partitions | Most recent transaction | After each transaction execution | Number of topic partitions involved in the last transaction. | ||||
Topic Info: TopicName | Partition Info:Partition ID | Latest Offset | When E1P is enabled | Offset value | Most recent transaction | After each transaction execution | Offset of the most recent message in this partition. | ||
Message Count | When E1P is enabled | Most recent transaction | Most recent transaction | After each transaction execution | Number of messages published to the partition since the start of the process. | ||||
Avg Message Size | When E1P is enabled | Average size in bytes | Most recent transaction | After each transaction execution | Average size of messages in the partition (in bytes), calculated as a rolling average. | ||||
Last Activity Timestamp | When E1P is enabled | Timestamp | Most recent transaction | After each transaction execution | Timestamp of the most recent write to the partition. | ||||
Total Kafka Messages Sent | When E1P is enabled | Count of messages | Most recent transaction | After each transaction execution | Total number of messages published to the topic across all partitions. | ||||
TOPIC_INFO | Topic Info: TopicName | Mapped Source Entity | Always | Source Entity name | Since the start of the application | When the first event is received for each topic. | In case of explicit and wildcard mapping, associated source entity name for the topic. In case of single topic mapping, ALL. | ||
Partition InfoPartition ID | Latest Offset | Always | Offset value | Since the start of the application | After each message sent / transaction execution | Offset of the most recent message in this partition. | |||
Message Count | Always | Count of messages | Since the start of the application | After each message sent / transaction execution | Number of messages published to the partition since the start of the application. | ||||
Avg Message Size | Always | Average size in bytes | Since the start of the application | After each message sent / transaction execution | Average size of messages in the partition (in bytes), calculated as a rolling average. | ||||
Last Activity Timestamp | Always | Timestamp | Since the start of the application | After each message sent / transaction execution | Timestamp of the most recent write to the partition. | ||||
Created By | When topic is created by adapter | Constant | Since the start of the application | On receiving first event for the topic | If a topic is created by the adapter during the current run of the application, the metric ‘Created By: STRIIM’ is added. Topics that were created in previous runs of the application are not displayed in the current application. | ||||
Total Kafka Messages Sent | Always | Count of messages | Since the start of the application | After each message sent / transaction execution | Total number of messages published to the topic across all partitions. |