Operational Considerations
For certificate roatation, see Certificate rotation for Kafka Writer.
Retries
Note the following regarding retry behavior:
When E1P is turned on, it guarantees no duplicates for both Internal and External retries
With A1P Internal Kafka retries will not cause any duplication or messages out of order if Producer is configured with following properties:
max.in.flight.requests.per.connection <= 5 enable.idempotence=true
When idempotence is set to true a producer id and sequence number will be assigned to each kafka message. This helps Kafka Producer clients to deduplicate messages. During external retries (Writer-level) a new Kafka producer will be created and all the messages will be retried. So we can have duplicates when there is a retry due to connection glitches or changes in connection information.
Refer to this link to understand how idempotence works in Kafka.
Monitoring
The writer supports detailed monitoring metrics as follows:
Metric name | Sub-metric mame | Sub-sub-metric name | Sub-sub-sub-metric name | Applicable/Visible when | Measure | Scope | Updated | Description |
|---|---|---|---|---|---|---|---|---|
No of Kafka messages sent | Always | Count of messages successfully published to Kafka | Since the application started | After each message sent (E1P=false)/ transaction execution(E1P=true) | Total number of messages successfully published to Kafka across all topics and partitions since the start of the application. | |||
Kafka Messages Rate | Always | Rate of messages published per second to Kafka | Since the application started | After each message sent (E1P=false)/ transaction execution(E1P=true) | Number of messages published to Kafka per second | |||
Kafka Write Rate | Always | Average bytes published per second to Kafka | Since the application started | After each message sent (E1P=false)/ transaction execution(E1P=true) | Average number of bytes sent per second to Kafka | |||
Discarded Event Count | Always | Count of events discarded | Since the application started | Whenever events are discarded | Total number of events that were discarded by the adapter. | |||
Buffered Messages Size In Bytes | Always | Current memory usage in bytes | Since the application started | Continuously updated - after receiving every event and after each message sent/transaction execution | Current memory usage for buffered messages/events in the adapter. | |||
Kafka Producer ID(s) | Always | Producer identifier | Since the application started | At initialization | ID of the Kafka Producer instance used for publishing messages. | |||
Average Latency | Always | Average time in milliseconds | Since the application started | After each transaction execution(E1P=true) or after each non transactional buffer execution(E1P=false) | Average time (in milliseconds) taken for Kafka request and acknowledgment per non-transactional buffer or transaction. | |||
Maximum Latency | Always | Maximum time in milliseconds | Since the application started | After each transaction execution(E1P=true) or after each non transactional buffer execution(E1P=false) | Maximum time (in milliseconds) taken for Kafka request and acknowledgment per non-transactional buffer or transaction. | |||
Last Send Timestamp | Always | Timestamp of send | Most recent Kafka successful Send API call (Transactional/ Non-Transactional) | After each transaction execution(E1P=true) or after each non transactional buffer execution(E1P=false) | Timestamp of the most recent successful Send operation. | |||
No. of Retries | Always | Count of retry attempts | Since the application started | When retry attempts are made | Total number of retry attempts made by the adapter due to connection or processing failures. | |||
Last Retry Attempt Timestamp | Always | Timestamp of the last retry attempt | Most recent retry | When retry attempts are made | The timestamp when the last retry attempt was made by the adapter. | |||
DDL Information | Schema Evolution | Always | Action type | Since the application started | At initialization | Action configured for DDL: Auto or Manual | ||
No. of DDLs Handled | Always | Count of DDLs processed | Since the application started | When DDL events are processed | Number of DDLs processed according to the configured Schema Evolution. | |||
Successful Transactions | When E1P is enabled | Count of transactions Committed | Since the application started | After each transaction execution(E1P=true) | Total number of transactions executed when transactional support is enabled. | |||
Average Messages per Transaction | When E1P is enabled | Average count of messages per transaction | Since the application started | After each transaction execution(E1P=true) | Average number of events in each committed transaction, calculated as a rolling average. | |||
Average Transaction Size In Bytes | When E1P is enabled | Average size in bytes | Since the application started | After each transaction execution(E1P=true) | Average size of messages in bytes of committed transactions, providing insight into transaction payload sizes. | |||
Max Transaction Size In Bytes | When E1P is enabled | Maximum size observed in bytes | Since the application started | After each transaction execution(E1P=true) | Maximum size in bytes of any committed transaction since the start of the process. | |||
Average Transaction Accumulation Time | When E1P is enabled | Average time in milliseconds | Since the application started | After each transaction execution(E1P=true) | Average time spent accumulating events for a transaction. | |||
Commit Lag | When E1P is enabled | Time difference in milliseconds | Since the application started | After each transaction execution(E1P=true) | Time taken between execution of a batch and its publication from each source, in milliseconds. | |||
Kafka Transactional ID(s) | When E1P is enabled | Identifier of the transactional producer | Since the application started | At initialization | ID of the Kafka’s Transactional Producer instance used for publishing messages. | |||
Last Transaction Information | No. of Events in the Transaction | When E1P is enabled | Count of events | Most recent transaction | After each transaction execution(E1P=true) | Number of events in the most recent committed transaction. | ||
Transaction size (in bytes) | When E1P is enabled | Size in bytes | Most recent transaction | After each transaction execution(E1P=true) | Size of the most recent transaction in bytes. | |||
Transaction Time (in ms) | When E1P is enabled | Time in milliseconds | Most recent transaction | After each transaction execution(E1P=true) | Time taken to complete the last Kafka I/O operation for the transaction. | |||
Transaction Accumulation Time (in ms) | When E1P is enabled | Time in milliseconds | Most recent transaction | After each transaction execution(E1P=true) | Time spent accumulating events before forming the batch for the last transaction. | |||
Topic Partition Count | When E1P is enabled | Count of partitions | Most recent transaction | After each transaction execution(E1P=true) | Number of topic partitions involved in the last transaction. | |||
Topic Info: TopicName | Partition Info:Partition ID | Latest Offset | When E1P is enabled | Offset value | Most recent transaction | After each transaction execution(E1P=true) | Offset of the most recent message in this partition. | |
Total No. of Messages | When E1P is enabled | Most recent transaction | Most recent transaction | After each transaction execution(E1P=true) | Number of messages published to this partition in the last transaction | |||
No. of DDLs | When E1P is enabled | Most recent transaction | Most recent transaction | After each transaction execution(E1P=true) | Number of DDL messages published to this partition in the last transaction | |||
Avg Message Size | When E1P is enabled | Average size in bytes | Most recent transaction | After each transaction execution(E1P=true) | Average size of messages published to the partition (in bytes) in the last transaction, calculated as a rolling average. | |||
Last Activity Timestamp | When E1P is enabled | Timestamp | Most recent transaction | After each transaction execution(E1P=true) | Timestamp of the most recent write to the partition. | |||
Total Kafka Messages Sent | When E1P is enabled | Count of messages | Most recent transaction | After each transaction execution(E1P=true) | Total number of messages published to the topic across all partitions in the last transaction. | |||
Total DDLs Sent | When E1P is enabled | Count of DDL Messages | Most recent transaction | After each transaction execution(E1P=true) | Total number of DDL messages published to the topic across all partitions in the last transaction. | |||
Buffer Memory in Bytes | When E1P is disabled | Current Buffer Memory in Bytes | Since the application started | After each event when E1P=false | Total size of data present in the Striim Buffer across all the topic partitions. | |||
Topic Information: TopicName | Mapped Source Entity | Always | Source Entity name | Since the start of the application | When the first event is received for each topic. | In case of explicit and wildcard mapping, associated source entity name for the topic. In case of single topic mapping, ALL. | ||
Partition InfoPartition ID | Latest Offset | Always | Offset value | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Offset of the most recent message in this partition. | ||
Total No. of Messages | Always | Count of messages | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Number of messages published to the partition since the start of the application. | |||
No. of DDLs | Always | Count of DDL messages | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Number of DDL messages published to the partition since the start of the application. | |||
Avg Message Size | Always | Average size in bytes | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Average size of messages published to the partition (in bytes)since the start of the application, calculated as a rolling average. | |||
Last Activity Timestamp | Always | Timestamp | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Timestamp of the most recent write to the partition. | |||
Created By | When topic is created by adapter | Constant | Since the start of the application | On receiving first event for the topic | If a topic is created by the adapter during the current run of the application, the metric ‘Created By: STRIIM’ is added. Topics that were created in previous runs of the application are not displayed in the current application. | |||
Total Kafka Messages Sent | Always | Count of messages | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Total number of messages published to the topic across all partitions. | |||
Total DDLs sent | Always | Count of DDL messages | Since the start of the application | After each message sent (E1P=false)/ transaction execution(E1P=true) | Total number of DDL messages published to the topic across all partitions. |