Testing KafkaWriter performance
Striim includes two utility scripts that may be useful in tuning your Kafka configuration. Striim/tools/enhanced-producer-perf.sh
writes to Kafka directly from the Striim host. Striim/tools/standalone-kafkawriter-perf.sh
writes with KafkaWriter.
Script arguments
The arguments are the same for both scripts:
argument | notes |
---|---|
file | The file to be used for testing. Specify |
input-format | If using a file, set to |
topic | The Kafka topic to write to. |
producer-props | A file containing the Kafka server properties. See |
output-format | Set to |
record-size | If using |
num-records | If using |
mode | Set to |
single-partition | Set to |
Script usage examples
For example, the following will generate one million 120-byte records and write them directly to Kafka.
./enhanced-producer-perf.sh --file none --topic test01 --producer-props props --record-size 120 --num-records 1000000
Output from that command would be similar to:
Configuration used for testing Apache Kafka Producer: File : none Topic : test01 Partitions : 1 Single Partition : false Mode : async Output format : none Num Records : 1000000 Record Size : 120 --------------------------------- Final Stats Records sent : 1000000 Records/sec : 521920.6680584551 MB/sec : 62.63048016701461 Avg Latency(ms) : 559.206501 Max Latency(ms) : 1054 Avg message size (bytes): 120.0 Number of times Producer send called : 1000000 Avg time between 2 Producer send calls(ms) : 0.003199727455364971 Max time between 2 Producer send calls(ms) : 210 ---------------------------------
The following will generate the same number of records of the same size but write them using KafkaWriter:
./standalone-kafkawriter-perf.sh --file none --topic test02 --producer-props props --record-size 120 --num-records 1000000
Output would be similar to:
Configuration used for testing Striim Kafka Writer: File : none Topic : test02 Partitions : 1 Single Partition : false Mode : async Output format : none Num Records : 1000000 Record Size : 120 --------------------------------- Final Stats Records sent : 1000000 Records/sec : 84918.47826086957 MB/sec : 3.9817331861413043 Avg Latency(ms) : 0.912115 Max Latency(ms) : 10018 Avg message size (bytes): 46.0 Number of times Producer send called : 1000000 Avg time between 2 Producer send calls(ms) : 0.011775 Max time between 2 Producer send calls(ms) : 10011