Kafka

Auto Map Partition

Boolean

True

When reading from multiple partitions, if there are multiple servers in the Striim deployment group on which KafkaReader is deployed ON ALL, partitions will be distributed automatically among the servers. Partitions will be rebalanced automatically as Striim servers are added to or removed from the group.

When deploying on a Forwarding Agent, set to False.

Broker Address

String

Specify the address of the Kafka broker in the format <IP address>:<port>.

Ignorable Exception

String

Available only in Kafka 0.11 and later versions.

You can specify a list of exceptions that Striim should ignore so as to avoid a system halt.

Currently, the only supported exception is "OFFSET_MISSING_EXCEPTION" which allows you to avoid a system halt when there is a partition purge. A partition purge is Kafka's process of freeing up Kafka message logs after the retention period expires. Messages have a TTL (time to live) with retention period properties in place. Upon expiry, messages are marked for deletion according to their creation timestamp. Once a message/offset is marked for deletion, and the consumer tries to read from the purged offset, Kafka throws an "OFFSET_MISSING_EXCEPTION".

Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE).

Kafka Config

String

See notes.

Specify any properties required by the authentication method used by the specified Kafka broker (see Configuring authentication in Kafka Config).

Optionally, specify Kafka consumer properties, separated by semicolons.

When reading from a topic in Confluent Cloud, specify the appropriate SASL properties.

When messages are in Confluent wire format, specify value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer.

When each message contains one Avro record and it is not length-delimited, set value.deserializer=com.striim.avro.deserializer.SingleRecordAvroRecordDeserializer.

The default is:

max.partition.fetch.bytes=10485760;
fetch.min.bytes=1048576;
fetch.max.wait.ms=1000;
receive.buffer.bytes=2000000;
poll.timeout.ms=10000;
request.timeout.ms=60001;
session.timeout.ms=60000;
value.deserializer=
  com.striim.avro.deserializer.LengthDelimitedAvroRecordDeserializer

Kafka Config Property Separator

String

;

Available only in Kafka 0.11 and later versions. Specify a different separator if one of the producer property values specified in KafkaConfig contains a semicolon.

Kafka Config Value Separator

String

=

Available only in Kafka 0.11 and later versions. Specify a different separator if one of the producer property values specified in KafkaConfig contains an equal symbol.

Partition ID List

String

Partition numbers to read from, separated by semicolons (for example, 0;1), or leave blank to read from all partitions.

Start Offset

Long

-1

With default value of -1, reads from the end of the partition. Change to 0 to read from the beginning of the partition.

If you specify startOffset, leave startTimestamp at its default value.

You can also specify multiple partition values for the Start Offset and Start Timestamp properties. Specify these values as a list of key-value pairs in the format "key=value; key=value". The keys and values are a list of partition IDs with an associated offset. For example:

StartOffset: '0=1; 1=0; 2=1024', where 0, 1, and 2 represent partition numbers, and the values represent the associated offsets.

Start Timestamp

String

If not specified, only new transactions (based on current Kafka host system time) are read. Specify a value in the format yyyy-MM-dd hh:mm:ss:SSS (for example, 2017-10-20 13:55:55.000) to start reading from an earlier point.

If the Kafka host and Striim host are not in the same time zone, specify the start time using the Striim host's time zone.

If you specify startTimestamp, leave startOffset at its default value.

You can also specify multiple partition values for the Start Offset and Start Timestamp properties. Specify these values as a list of key-value pairs in the format "key=value; key=value". The keys and values are a list of partition IDs with an associated timestamp. For example:

StartTimestamp: '0=<TS_0>;1=<TS_2>;2=<TS_3>', where 0, 1, and 2 represent partition numbers, and the values represent the associated start timestamps.

Topic

String

Do not modify this property when recovery is enabled for the application.