Upgrading Existing Applications that use Kafka Writer
This section covers what you need to know if you have Kafka Writer applications in Striim 5.2 or earlier and are upgrading or have already upgraded to Striim 5.4.
Notable Changes When Upgrading to Striim 5.4
The new Kafka Writer mandates the use of a Kafka Connection Profile.
Existing Kafka Writers will be seamlessly upgraded. Connection, producer, and consumer configurations in the old KafkaConfig property are parsed and migrated to a Kafka Connection Profile.
Kafka Connection Profiles are created and linked to each Kafka Writer instance automatically.
Backward Compatibility
When upgraded from Striim 5.2 or earlier, the following behaviors continue as before when PreserveBackwardCompatibility is TRUE (default after upgrade):
StriimMetadata (with a null value) is added to messages when E1P is true.
Message Keys are JSON Strings.
Avro schemas are created based on Striim Type for both DML and DDL changes.
Special characters are replaced using the mangled name format.
To migrate to new Kafka Writer behavior, set PreserveBackwardCompatibility to FALSE.
Note
If you import TQL exported from Striim 5.2 or earlier, Kafka Reader and Kafka Writer will use the deprecated Kafka 2.1 version. To use the new adapters, edit the TQL before importing.
Legacy Kafka Writer 2.1 Limitations with Kafka 4.0+
While the legacy Kafka Writer 2.1 can technically connect to Kafka 4.0 brokers, it has significant limitations:
Cannot use the new consumer group protocol (KIP-848).
Cannot access Kafka 4.0+ features.
Operates at minimum supported protocol versions.
Potential performance degradation.
Note
Apache Kafka 4.0 removed support for client protocol versions older than 2.1 --- the legacy Kafka Writer 2.1 is incompatible with Kafka 4.0+ brokers.
Enhanced Features in New Kafka Writer
Schema compatibility defaults to NONE but can be altered.
Subject Name mapping defaults to standard strategy, with options UseTopicName or UseDynamicValues for WAEvents.
KafkaWriter Changes in 5.4
All connection information comes from the connection profile (supports vault extraction and certificate rotation while running).
Topics property accepts a single topic name or source entity to topic mappings.
Topics can be created automatically if configured.
Message Keys can be Custom or Primary Keys.
Events can be distributed to multiple partitions based on a custom partition key or Message Key.
E1P toggle (default: true) controls message semantics.
E1P requires an additional checkpointing topic.
Both Striim and Confluent serialization formats supported via the Serializer property.
Property Comparison Table
Kafka Writer 2.1 property | KafkaWriter (5.4) property |
|---|---|
(none — new property) | PreserveBackwardCompatibility: True |
Mode: Sync (with recovery ON) | E1P: True (with recovery ON) |
Mode: Async | E1P: False (reflects the recovery setting) |
Topic | Topics |
BrokerAddress, KafkaConfig | ConnectionProfileName (Endpoint Type -- Kafka) |
MessageKey -- empty | MessageKey: None |
MessageKey | MessageKey: Custom, CustomMessageKey |
MessageHeader | MessageHeader |
Partition Key -- empty | PartitionKey: None |
Partition Key | PartitionKey: Custom, CustomPartitionKey |
Parallel Threads | Parallel Threads |
value.serializer: Confluent (set via KafkaConfig) | Serializer: ConfluentSerializer |
The Async Producer API is used regardless of the E1P setting, with producer configuration adjusted to match the chosen message semantics.
Avro Formatter Changes in 5.4
Message key serialization with the Avro Formatter is now the default. Schema generation and evolution are tracked via Schema Registry only --- SchemaFileName is not supported.
Schema Compatibility setting can be configured via AvroFormatter.
Subject naming: in version 2.1.0, subject name mapping was supported only for AvroEvent and JsonNodeEvent types. After upgrade, UseTopicName and UseDynamicValues are also supported for WAEvent in Native, Table, and Default formatAs modes.
Schema tracking is supported only via the Schema Registry.
Message Structure Changes in 5.4
Message Key Format
In the Avro Formatter, message keys are now Avro records that maintain original data types, with separate schemas registered. Backward compatibility mode retains JSON String keys.
StriimMetadata
StriimMetadata has been eliminated from message overhead in the new Kafka Writer. A checkpointing topic is used instead. Backward compatibility mode retains the StriimMetadata field (value always null).
Avro Serialization Changes in 5.4
Enhanced Datatype Mapping: Field names and types are now based on the Schema Conversion Tool output when the source schema flows through the pipeline.
Schema Compatibility: Default remains None; other compatibility modes are now configurable.
Aliases: No longer maintained in the new Kafka Writer. Source field names are directly mapped as Avro field names. Backward compatibility mode retains uppercase field names with actual names as aliases.
Resource Consumption Changes in 5.4
Exactly-once processing uses Kafka Transactions together with Striim's proprietary checkpointing.
Old overhead eliminated: no StriimMetadata in messages, no sync producer dependency, no batching of multiple events into a single Kafka message.
A checkpointing topic is created if not already present (requires topic creation permissions).
TQL Examples Before and After Upgrade
For additional TQL patterns covering topic mapping, partitioning, and serialization on the new Kafka Writer, see Sample TQLs for Various Scenarios.
Before (Striim 5.2 TQL)
CREATE OR REPLACE TARGET KAFKA_MTLS_AVRO USING Global.KafkaWriter VERSION '2.1.0'( KafkaConfig: 'request.timeout.ms==60001:session.timeout.ms==60000:security.protocol==SSL:...', MessageKey: 'PatientHistoryFrom=@metadata(TableName)', MessageHeader: 'HospitalName=\"UniversalHealthCare\"', Mode: 'ASync', Topic: 'PATIENT_RECORDS_AVRO', brokerAddress: 'pkc-nyn9x6.us-east1.gcp.confluent.cloud:9092' ) FORMAT USING Global.AvroFormatter ( formatAs: 'Native', ... ) INPUT FROM ORACLE_OP_STREAM;
After (Striim 5.4 Converted TQL)
Upgrading this application creates a Kafka Connection Profile holding the mutual TLS connection details that were previously embedded in KafkaConfig:
Kafka Connection Profile after upgrade
CREATE OR REPLACE TARGET KAFKA_MTLS_AVRO USING Global.KafkaWriter ( adapterName: 'KafkaWriter', CustomMessageKey: 'PatientHistoryFrom=@metadata(TableName)', MessageKey: 'Custom', MessageHeader: 'HospitalName=\"UniversalHealthCare\"', E1P: false, Topics: 'PATIENT_RECORDS_AVRO', PartitionKey: 'None', Serializer: 'ConfluentSerializer', connectionProfileName: 'admin.KAFKA_MTLS_AVRO_CP', PreserveBackwardCompatibility: true ) FORMAT USING Global.AvroFormatter ( formatAs: 'Native', ... ) INPUT FROM ORACLE_OP_STREAM;
Connection-related properties are moved to the new connection profile.