Skip to main content

Example Use Cases

The following scenarios illustrate how the Kafka Writer's key considerations are applied to meet specific business and operational requirements.

Use Case 1. Global Compliance Audit Stream

Single Topic · Single Partition

Business Context

A regulated enterprise must maintain a complete, ordered record of audit events for compliance, investigations, and replayability.

Target Design Goals

  • Global ordered sequence of events

  • Deterministic replay

  • Strong schema governance

Configuration Guidance

  • Topic and Partitioning: All events are written to a single topic with one partition to preserve strict global ordering.

  • Delivery Semantics: Enable Exactly Once Processing (E1P) to avoid message duplication during retries and to support snapshot-consistent replay.

  • Message Key: Message keys may be omitted or derived from audit categories for consumer filtering; ordering is maintained through the single partition.

  • Serializer: Use Avro with Schema Registry to enforce schema contracts, ensure compatibility, and support audit lifecycle governance.

  • Headers: Include metadata headers such as eventCategory, sourceSystem, and severity to aid filtering in downstream tools without payload overhead.

This configuration ensures audit integrity and supports drift-free schema evolution.

Use Case 2. Real-Time Clickstream Analytics

Single Topic · Multiple Partitions

Business Context

An online platform ingests high-velocity user interaction events for analytics, personalization, and monitoring.

Target Design Goals

  • Scale ingestion and consumption

  • Preserve event order per user/session

  • Support real-time aggregation and stateful computation

Configuration Guidance

  • Partition Key: Choose a field such as sessionId or userId to preserve order for that logical grouping while enabling parallelism across many sessions.

  • Message Key: Use a custom message key corresponding to the partition key (e.g., sessionId), enabling stateful processing and downstream correlation.

  • Serializer: JSON may be used for flexibility; Avro can be selected when downstream analytics enforce contracts or schema evolution must be controlled.

  • Delivery Semantics: At Least Once Processing (A1P) is acceptable when occasional duplicates can be deduplicated by consumers.

  • Headers: Include session metadata (e.g., pageCategory, deviceType) for real-time filter routing.

This configuration balances scale, localized ordering, and flexible downstream analytics.

Use Case 3. OLTP CDC Replication by Entity

Multiple Topics · Single Partition per Topic

Business Context

A data platform replicates change data capture streams from OLTP systems into Kafka to feed microservices, reporting, and analytics.

Target Design Goals

  • Logical separation of entities

  • Entity-level ordering

  • Schema evolution tracking

  • Stateful downstream joins and aggregations

Configuration Procedure

  1. Topic Mapping: Map each source entity (e.g., orders, customers, payments) to its own topic.

  2. Partitioning Strategy: Use one partition per topic to preserve deterministic ordering within that entity's stream.

  3. Message Key: Configure the message key using the source table's primary key to enable log compaction and efficient stateful processing.

  4. Serializer: Use Avro with Schema Registry to ensure schema evolution is governed and compatible across consumers.

  5. Delivery Semantics: Enable Exactly Once Processing (E1P) for transactional correctness and duplicate elimination.

  6. Headers: Add fields such as entityName and operationType for downstream routing and auditing.

This configuration supports strong entity isolation, ordered replication, and governed schema evolution.

Use Case 4. IoT Telemetry by Device Category

Multiple Topics · Multiple Partitions

Business Context

An industrial IoT platform ingests telemetry and alert events from distributed devices across categories such as sensor readings, alert notifications, and operational states.

Target Design Goals

  • High-volume ingestion

  • Category-level logical separation

  • Localized ordering by device or region

  • Efficient filtering and routing

Configuration Guidance

  • Topic Strategy: Route telemetry to separate topics based on device category to support independent downstream processing SLAs.

  • Partition Key: Use fields such as deviceId or regionId to group related events within partitions, preserving local ordering while enabling scale.

  • Message Key: Configure a key based on deviceId to support compaction where appropriate and to aid stateful processing.

  • Serializer: JSON offers flexibility; Avro is appropriate where schema evolution must be governed across device types.

  • Delivery Semantics: At Least Once Processing (A1P) for most telemetry; select E1P for high-priority alert streams.

  • Headers: Add metadata such as severity, batteryLevel, or region to support efficient stream routing and monitoring.

This configuration accommodates high throughput while preserving localized ordering and metadata-based filtering.

Summary

The Kafka Writer's configurability around partition keys, message keys, serializers, delivery semantics, headers, and retry behavior makes it a comprehensive solution for diverse Kafka pipeline requirements. By starting from business goals and mapping them to these configuration dimensions, you can build pipelines that balance ordering, scale, governance, and operational resilience.