Skip to main content

Key Considerations

The Kafka Writer supports a range of configuration dimensions that must be considered together when designing a pipeline.

Topic Strategy (Single Topic vs Multiple Topics)

The first design decision is how events should be logically organized in Kafka.

Single Topic

Use a single topic when:

  • Events represent one logical stream

  • Consumers subscribe to the entire dataset

  • Separation by entity or category is unnecessary

  • Global replay or unified analytics is required

Examples include:

  • Compliance audit logs

  • Unified clickstream feeds

A single topic simplifies subscription models but may require partitioning for scale.

Multiple Topics

Use multiple topics when:

  • Events belong to distinct entities or business domains

  • Consumers subscribe selectively to subsets of data

  • Different SLAs or retention policies are required

  • Logical separation improves clarity or governance

Examples include:

  • CDC replication of customers, orders, and payments

  • IoT telemetry separated by device category

Separating topics improves isolation, governance, and operational control.

Partition Key

The partition key determines which partition a message is written to. It is critical for:

  • Localized ordering --- preserving order within a defined key group (e.g., userId, sessionId, deviceId)

  • Load distribution --- evenly spreading events across partitions for scale

  • Co-location for stateful processing --- enabling consumers to maintain local state per key

Choosing an appropriate partition key depends on workload characteristics. For instance, session-level partitioning is essential for clickstream analytics to ensure events for a given session remain ordered while still parallelizing across many sessions.

Message Key

The message key influences:

  • Compaction semantics --- keys used in compacted topics retain only the latest message per key

  • Consumer grouping and join strategies --- enabling consumers to perform keyed aggregations or joins efficiently

  • Ordering within partitions --- messages with the same key land in the same partition, preserving order

Deriving message keys from business identifiers such as primary keys or natural keys (e.g., orderId, customerId) makes topics more meaningful and enables stateful processing in downstream systems.

Serializer Selection

The serializer defines how events are encoded:

  • JSON: Provides a flexible and lightweight serialization format suitable for environments where schema evolution is loosely governed or where consumers have diverse and independent processing requirements. JSON is commonly used when rapid iteration and minimal schema enforcement are preferred.

  • Avro (with Schema Registry): Provides a schema-aware serialization mechanism that enables controlled schema evolution and compatibility validation. When integrated with Schema Registry, Avro allows producers and consumers to enforce compatibility rules, ensuring that changes to data structures do not disrupt downstream applications. This approach is recommended for enterprise environments requiring governance, contract management, and long-term interoperability.

Choosing the right serializer impacts schema governance, consumer evolution, and compatibility across diverse clients.

Delivery Semantics

  • Exactly Once Processing (E1P) - Leverages Kafka transactional producers and checkpointing to eliminate duplicates and provide atomicity.

  • At Least Once Processing (A1P) - Prioritizes throughput and simplicity, accepting potential duplicates that may be resolved downstream.

Headers and Metadata

Message headers allow you to carry lightweight routing or metadata without modifying the message body. This supports:

  • Filtering on metadata (e.g., severity, eventType)

  • Consumer routing without parsing full payloads

Retry and Recovery

The writer supports both Kafka internal retries and writer-level retries. Combined with E1P, this ensures resilience and correctness in the face of transient failures. See Retries and Error Recovery for troubleshooting guidance.