Kafka Writer

Note

Kafka Writer versions 0.11 and 2.1.1 are no longer supported. As part of upgrading to Striim 5.4.x, all Kafka Writer adapters in created or quiesced applications will be upgraded to the new versions of the adapter and corresponding connection profiles will be created. If after upgrading you need to import TQL exported from Striim 5.2 or earlier, edit the TQL before importing to use the new properties, and create a connection profile with the appropriate properties. For more information, see Upgrading Existing Applications that use Kafka Writer.

Kafka Writer supports writing to one or more topics in a Kafka cluster in Apache Kafka, Amazon MSK, Confluent Cloud, or Confluent Platform versions 2.0.0 to 4.0.1.

Kafka SmartMap Overview

What is Kafka SmartMap?

Kafka SmartMap is Striim's high-performance Kafka target capability for streaming data from Striim-supported sources into Apache Kafka and Kafka-compatible platforms, including Confluent Platform, Confluent Cloud, and Amazon MSK.

Kafka SmartMap writes initial load, change data, and event streams from sources such as relational databases, NoSQL databases, data warehouses, files, SaaS applications, message buses, and application adapters into Kafka topics. It is designed to make Kafka pipelines easier to configure and operate by supporting source-to-topic mapping, topic creation, schema creation and evolution, secure connection profiles, and configurable delivery semantics.

Kafka SmartMap does not perform change capture by itself. Instead, it writes data captured or produced by upstream Striim sources, including CDC sources, into Kafka with the topic structure, message format, keys, partitioning strategy, and reliability guarantees required by downstream consumers.

Why Kafka SmartMap?

Kafka is often used as the real-time distribution layer between operational systems and downstream applications. However, writing all source changes into a single topic can make downstream processing more complex. Consumers may need to filter irrelevant events, coordinate with other teams, or handle mixed schemas and workloads in one stream.

Kafka SmartMap simplifies this model by preserving source-level separation when needed. Each source table, object, collection, or event category can be mapped to its own Kafka topic, while related streams can also be consolidated into a shared topic when that better matches the downstream use case.

This provides several benefits:

Selective consumption: downstream teams subscribe only to the topics they need.
Cleaner data products: topics can align with business entities such as customers, orders, payments, devices, or audit events.
Better scalability and isolation: a spike in one source entity or topic is less likely to affect unrelated consumers.
Simpler downstream logic: consumers do not need to filter a single mixed stream before processing.
Governed schemas: Avro and schema registry integration help manage data contracts and schema evolution.
Flexible reliability: exactly-once processing supports high-integrity workloads, while at-least-once processing supports higher-throughput use cases where downstream deduplication is acceptable.
Enterprise connectivity: connection profiles centralize Kafka connectivity, authentication, encryption, and credential management.

For example, a retail enterprise may stream inventory, order, customer, and payment changes into Kafka so fulfillment, marketing, analytics, and customer-facing applications can react in real time. If the pipeline reads from a replica, secondary database, warehouse, lake, or another upstream source rather than directly from the production database, it can reduce load on the primary system while still keeping downstream services current.

Kafka SmartMap summary

Supported sources	Kafka SmartMap can write data from all sources supported by Striim.
Supported targets	Kafka SmartMap writes to one or more topics in a Kafka cluster on: Apache Kafka Amazon Managed Streaming for Apache Kafka (MSK) Confluent Cloud Confluent Platform Supports Kafka broker versions 2.0.0 through 4.0.1
Authentication	Authenticates its connection to Kafka using SASL (PLAIN, GSSAPI/Kerberos, SCRAM-256, SCRAM-512) or mutual TLS, with SSL used to encrypt the connection. Kafka Connection Profiles can use Striim Vault to manage credentials and certificates.
Supported formatters	Supports Avro, DSV, JSON, and XML message formats: Avro Formatter: AvroEvent, JsonNodeEvent, ParquetEvent, WAEvent DSV Formatter: WAEvent JSON Formatter: JsonNodeEvent, WAEvent XML Formatter: WAEvent
Supported topic and partition mapping	Supports writing different combinations of source table, file, topic, or document to Kafka topics: Single topic with one partition Single topic with multiple partitions Multiple topics with single partition each Multiple topics with multiple partitions Topics can be pre-created or created automatically if they do not exist.
Additional writing features	Supports Auto Detection of Primary Key Columns for Database Sources or custom message keys and partition keys, custom Kafka message headers, and schema evolution using Confluent Schema Registry or Karapace.
Resilience and recovery	Supports connection retry to avoid application halting due to transient connection issues. Supports recovery with exactly-once processing (E1P), using Kafka transactions and Striim checkpointing, or at-least-once processing (A1P).
Performance	Supports parallel threads to increase throughput when writing to multiple topics.
Programmability	Flow Designer, TQL, and wizards in the web UI to create applications from supported sources.
Metrics and auditing	Key metrics are available through Striim's monitoring features.
Java client version	Kafka SmartMap uses Kafka client version 3.9.1, and supports Kafka broker versions 2.0.0 through 4.0.1 on Apache Kafka, Amazon MSK, Confluent Cloud, and Confluent Platform. Kafka SmartMap requires JDK 11 or JDK 17.
Additional Information	The upper bound of the Kafka broker version matters, specifically for Kafka 4.0+ compatibility: Apache Kafka 4.0 removed support for client protocol versions older than 2.1, so earlier Kafka Writer releases (client 2.1) cannot use the new consumer group protocol (KIP-848) or other 4.0+ features against a Kafka 4.0+ broker, and may see reduced performance. Use Kafka SmartMap to connect to Kafka 4.0+ brokers.