Skip to main content

Kafka Writer

Note

Kafka Writer versions 0.11 and 2.1.1 are no longer supported. As part of upgrading to Striim 5.4, all Kafka Writer adapters in created or quiesced applications will be upgraded to the new versions of the adapter and corresponding connection profiles will be created. If after upgrading you need to import TQL exported from Striim 5.2 or earlier, edit the TQL before importing to use the new properties, and create a connection profile with the appropriate properties. For more information, see Kafka Writer upgrade guide.

Striim can write to Apache Kafka using its Kafka Writer. Kafka Writer uses Kafka client version 3.9.1, and supports writing to one or more topics in a Kafka cluster in Apache Kafka, Amazon MSK, Confluent Cloud, or Confluent Platform versions 2.0.0 to 4.0.1.

Among other improvements, in this release, Kafka Writer:

  • Supports writing different combinations of source table, file, topic, or document to Kafka topics:

    • single topic with one partition

    • single topic with multiple partitions

    • multiple topics with single partition each

    • multiple topics with multiple partitions

  • Can write to existing topics or create them automatically if they do not exist.

  • Supports SASL and mTLS based authentication methods as well SSL-based encryption for data security..

  • Uses connection profiles for connecting to Kafka and the Confluent schema registry.

  • Supports multiple input stream types:

    • using Avro Formatter: AvroEvent, JsonNodeEvent, ParquetEvent, WAEvent

    • using DSV Formatter: WAEvent

    • using JSON Formatter: JsonNodeEvent,WAEvent

    • using XML Formatter: WAEvent

  • Supports Avro, DSV, JSON, and XML message formats.

  • Supports recovery with both E1P (exactly once processing) or A1P (at least once processing.

  • Supports parallel writes to multiple topics for higher throughput.