Skip to main content

Key Concepts

Kafka SmartMap builds on standard Apache Kafka terminology. This section defines the core terms used throughout this documentation for readers who are new to Kafka. For guidance on how to apply these concepts when designing a pipeline, see Key Considerations below.

  • Topic --- A named, append-only log that Kafka stores and partitions. Producers write records to a topic; consumers read from it. Kafka Writer maps source data into one or more topics based on the topic strategy in effect (see Topic Strategy above).

  • Partition --- An ordered, immutable subdivision of a topic. Kafka guarantees message order only within a single partition, not across an entire topic. Kafka Writer's partition strategy determines how records are distributed across a topic's partitions.

  • Consumer group --- A set of consumers that cooperate to read a topic, with each partition consumed by only one member of the group at a time. Consumer groups let downstream applications scale out horizontally while each partition's messages are still processed in order by a single consumer.

  • Schema registry --- A centralized service (such as Confluent Schema Registry or Karapace) that stores and versions Avro schemas for topics and enforces compatibility rules between schema versions. Kafka Writer can register and evolve schemas automatically when AvroFormatter is configured with SchemaEvolution: Auto (see Connecting to Schema Registry and Schema Evolution).

  • Connection profile --- A reusable, centrally managed set of connection settings --- broker addresses, authentication credentials, schema registry connection details, and security configuration --- that a Kafka Writer TQL application references by name (ConnectionProfileName) instead of repeating inline (see Kafka Connection Profile Properties).

  • Commit policy --- The setting that controls how often Kafka Writer commits (flushes and checkpoints) a batch of events, expressed as an event count and/or a time interval, for example EventCount=10000;Interval=15s. It governs the size of each transactional batch under E1P (see Kafka Writer Properties).

  • E1P (Exactly-once processing) --- A delivery semantic in which Kafka Writer uses Kafka transactions and Striim checkpointing to guarantee each event is written exactly once, even across retries and restarts. See E1P --- Exactly-Once Processing for full configuration details.

  • A1P (At-least-once processing) --- A delivery semantic that prioritizes throughput and operational simplicity over duplicate elimination; events may be written more than once during retries, and duplicate handling is left to downstream consumers. See A1P --- At-Least-Once Processing for full configuration details.

How Kafka SmartMap Works

Designing a Kafka SmartMap pipeline is best understood as a sequence of five decisions. Each decision maps a business requirement, such as ordering, scale, schema governance, or duplicate handling, to a concrete Kafka configuration.

image2.png

Designing a Kafka SmartMap pipeline: five decisions

1. Topic Strategy

The first decision is how source data should be organized into Kafka topics.

Kafka SmartMap supports several common topic-mapping patterns:

image4.png

Kafka SmartMap topic mapping patterns

Use one topic per source entity when consumers need clean separation, independent subscriptions, or different retention and processing behavior. Use event routing when a single source stream contains different categories of events that should be consumed separately. Use consolidation when several sources share a common schema and downstream consumers need a unified stream.

2. Partition Strategy

Partitions determine how data is distributed within a topic.

A single partition preserves strict ordering for all events in a topic, which is useful for low-volume control streams, audit streams, or ordered replication scenarios. Multiple partitions increase throughput and allow consumers to process events in parallel, but ordering is guaranteed only within each partition.

The right partition strategy depends on whether the workload prioritizes global ordering, localized ordering, or scale.

3. Keying Strategy

Keys determine how Kafka groups and routes records.

Kafka SmartMap can use message keys and partition keys derived from primary keys, source fields, metadata, or user data. For CDC workloads, using the source table primary key as the message key helps keep all changes for the same row associated with the same logical key. This supports downstream use cases such as compaction, joins, aggregations, and stateful processing.

When message-key-based partitioning is used, related events can be routed to the same partition so consumers process them in order within that key group.

4. Data Representation

Data representation determines how Kafka messages are encoded and how schemas are managed.

Kafka SmartMap supports flexible message formats such as JSON, DSV, XML, and Avro. JSON is useful for rapid development, broad interoperability, and flexible consumers. Avro is recommended for governed enterprise Kafka pipelines, especially CDC pipelines, because it supports compact encoding, schema registration, compatibility checks, and schema evolution.

When Avro is used with a schema registry, Kafka SmartMap can help manage schema creation, schema tracking, and schema evolution so producers and consumers continue to share a reliable data contract as source structures change.

5. Delivery Semantics

Delivery semantics determine how the pipeline behaves during retries, restarts, and failures.

Kafka SmartMap supports:

  • Exactly-once processing (E1P): uses Kafka transactions and Striim checkpointing to avoid duplicate writes. This is recommended for workloads that require transactional correctness, such as CDC replication, financial data, and trusted operational streams. See E1P --- Exactly-Once Processing for configuration details.

  • At-least-once processing (A1P): prioritizes throughput and operational simplicity while allowing occasional duplicates that can be handled downstream. This is useful for high-volume telemetry, clickstream, and log-style workloads. See A1P --- At-Least-Once Processing for configuration details.

Choosing the right delivery mode helps balance correctness, latency, throughput, and recovery behavior.

Next Steps

The rest of this documentation explains how to configure these decisions in Kafka SmartMap.