Key Considerations
The Kafka Writer supports a range of configuration dimensions that must be considered together when designing a pipeline.
Topic Strategy (Single Topic vs Multiple Topics)
The first design decision is how events should be logically organized in Kafka.
Single Topic
Use a single topic when:
Events represent one logical stream
Consumers subscribe to the entire dataset
Separation by entity or category is unnecessary
Global replay or unified analytics is required
Examples include:
Compliance audit logs
Unified clickstream feeds
A single topic simplifies subscription models but may require partitioning for scale.
Multiple Topics
Use multiple topics when:
Events belong to distinct entities or business domains
Consumers subscribe selectively to subsets of data
Different SLAs or retention policies are required
Logical separation improves clarity or governance
Examples include:
CDC replication of customers, orders, and payments
IoT telemetry separated by device category
Separating topics improves isolation, governance, and operational control.
Partition Key
The partition key determines which partition a message is written to. It is critical for:
Localized ordering --- preserving order within a defined key group (e.g., userId, sessionId, deviceId)
Load distribution --- spreading events across partitions for scale
Co-location for stateful processing --- enabling consumers to maintain local state per key
Choosing an appropriate partition key depends on workload characteristics. For instance, session-level partitioning is essential for clickstream analytics to ensure events for a given session remain ordered while still parallelizing across many sessions.
Message Key
The message key influences:
Compaction semantics --- keys used in compacted topics retain only the latest message per key
Consumer grouping and join strategies --- enabling consumers to perform keyed aggregations or joins efficiently
Ordering within partitions --- messages with the same key land in the same partition, preserving order
Deriving message keys from business identifiers such as primary keys or natural keys (e.g., orderId, customerId) makes topics more meaningful and enables stateful processing in downstream systems.
Serializer Selection
The serializer defines how events are encoded:
JSON: Provides a flexible and lightweight serialization format suitable for environments where schema evolution is loosely governed or where consumers have diverse and independent processing requirements. JSON is commonly used when rapid iteration and minimal schema enforcement are preferred.
Avro (with Schema Registry): Provides a schema-aware serialization mechanism that enables controlled schema evolution and compatibility validation. When integrated with Schema Registry, Avro allows producers and consumers to enforce compatibility rules, ensuring that changes to data structures do not disrupt downstream applications. This approach is recommended for enterprise environments requiring governance, contract management, and long-term interoperability.
DSV and XML formats are also supported.
Choosing the right serializer impacts schema governance, consumer evolution, and compatibility across diverse clients.
Delivery Semantics
Exactly Once Processing (E1P): Leverages Kafka transactional producers and checkpointing to eliminate duplicates and provide atomicity.
At Least Once Processing (A1P): Prioritizes throughput and simplicity, accepting potential duplicates that may be resolved downstream.
Headers and Metadata
Message headers allow you to carry lightweight routing or metadata without modifying the message body. This supports:
Filtering on metadata (e.g., severity, eventType)
Consumer routing without parsing full payloads
Retry and Recovery
The writer supports both Kafka internal retries and writer-level retries. Combined with E1P, this ensures resilience and correctness in the face of transient failures. See Retries and Error Recovery for troubleshooting guidance.
Example Use Cases
The following scenarios illustrate how the Kafka Writer's key considerations are applied to meet specific business and operational requirements.
Use Case 1. Global Compliance Audit Stream
Single Topic · Single Partition
Business Context
A regulated enterprise must maintain a complete, ordered record of audit events for compliance, investigations, and replayability.
Target Design Goals
Global ordered sequence of events
Deterministic replay
Strong schema governance
Configuration Guidance
Topic and Partitioning: All events are written to a single topic with one partition to preserve strict global ordering.
Delivery Semantics: Enable Exactly Once Processing (E1P) to avoid message duplication during retries and to support snapshot-consistent replay.
Message Key: Message keys may be omitted or derived from audit categories for consumer filtering; ordering is maintained through the single partition.
Serializer: Use Avro with Schema Registry to enforce schema contracts, ensure compatibility, and support audit lifecycle governance.
Headers: Include metadata headers such as eventCategory, sourceSystem, and severity to aid filtering in downstream tools without payload overhead.
This configuration ensures audit integrity and supports drift-free schema evolution.
Use Case 2. Real-Time Clickstream Analytics
Single Topic · Multiple Partitions
Business Context
An online platform ingests high-velocity user interaction events for analytics, personalization, and monitoring.
Target Design Goals
Scale ingestion and consumption
Preserve event order per user/session
Support real-time aggregation and stateful computation
Configuration Guidance
Partition Key: Choose a field such as sessionId or userId to preserve order for that logical grouping while enabling parallelism across many sessions.
Message Key: Use a custom message key corresponding to the partition key (e.g., sessionId), enabling stateful processing and downstream correlation.
Serializer: JSON may be used for flexibility; Avro can be selected when downstream analytics enforce contracts or schema evolution must be controlled.
Delivery Semantics: At Least Once Processing (A1P) is acceptable when occasional duplicates can be deduplicated by consumers.
Headers: Include session metadata (e.g., pageCategory, deviceType) for real-time filter routing.
This configuration balances scale, localized ordering, and flexible downstream analytics.
Use Case 3. OLTP CDC Replication by Entity
Multiple Topics · Single Partition per Topic
Business Context
A data platform replicates change data capture streams from OLTP systems into Kafka to feed microservices, reporting, and analytics.
Target Design Goals
Logical separation of entities
Entity-level ordering
Schema evolution tracking
Stateful downstream joins and aggregations
Configuration Procedure
Topic Mapping: Map each source entity (e.g., orders, customers, payments) to its own topic.
Partitioning Strategy: Use one partition per topic to preserve deterministic ordering within that entity's stream.
Message Key: Configure the message key using the source table's primary key to enable log compaction and efficient stateful processing.
Serializer: Use Avro with Schema Registry to ensure schema evolution is governed and compatible across consumers.
Delivery Semantics: Enable Exactly Once Processing (E1P) for transactional correctness and duplicate elimination.
Headers: Add fields such as entityName and operationType for downstream routing and auditing.
This configuration supports strong entity isolation, ordered replication, and governed schema evolution.
Use Case 4. IoT Telemetry by Device Category
Multiple Topics · Multiple Partitions
Business Context
An industrial IoT platform ingests telemetry and alert events from distributed devices across categories such as sensor readings, alert notifications, and operational states.
Target Design Goals
High-volume ingestion
Category-level logical separation
Localized ordering by device or region
Efficient filtering and routing
Configuration Guidance
Topic Strategy: Route telemetry to separate topics based on device category to support independent downstream processing SLAs.
Partition Key: Use fields such as deviceId or regionId to group related events within partitions, preserving local ordering while enabling scale.
Message Key: Configure a key based on deviceId to support compaction where appropriate and to aid stateful processing.
Serializer: JSON offers flexibility; Avro is appropriate where schema evolution must be governed across device types.
Delivery Semantics: At Least Once Processing (A1P) for most telemetry; select E1P for high-priority alert streams.
Headers: Add metadata such as severity, batteryLevel, or region to support efficient stream routing and monitoring.
This configuration accommodates high throughput while preserving localized ordering and metadata-based filtering.
Summary
The Kafka Writer's configurability around partition keys, message keys, serializers, delivery semantics, headers, and retry behavior makes it a comprehensive solution for diverse Kafka pipeline requirements. By starting from business goals and mapping them to these configuration dimensions, you can build pipelines that balance ordering, scale, governance, and operational resilience.