Skip to main content

Kinesis Writer

Writes to an Amazon Kinesis stream.

Limitations:

  • Only sync mode is supported.

  • Parallel threads are not supported.

  • Do not write to a target stream with more than 249 shards.

  • Do not merge, split, start or stop encryption on, or change the shard count of the target stream.

  • QUIESCE is not supported in this release.

  • If multiple writers write to the same shard, there may be duplicate events after Recovering applications (in other words, exactly-once processing cannot be guaranteed).

  • The maximum record length cannot exceed 1MB, including the metadata Striim adds to support recovery. (This is a limitation in Kinesis, not Striim.)

Kinesis Writer properties

property

type

default value

notes

Access Key ID

String

Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with write permission on the stream.

When Striim is running in Amazon EC2 and there is an IAM role with that permission associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role.

Batch Policy

String

Size:900000, Interval:1

The batch policy includes size and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to Size:-1.

With the default setting, data will be written every second or sooner if the an event pushes the buffer past 900,000 bytes. The buffer will expand as necessary to include that last event in the batch.

Connection Retry Policy

String

retries:5, retryBackOff:1000ms

With the default retries:5, retryBackOff:1000ms, if the first connection attempt is unsuccessful, Striim will try again every one second. If the fifth attempt is unsuccessful, the adapter will halt and log an exception. Negative values are not supported.

Mode

String

Sync

Do not change from default.

Partition Key

String

Optionally, specify a field to be used to partition the events among multiple shards. See Kafka Writer for more details.

Region Name

String

the AWS region of the stream (see AWS Regions and Endpoints)

Secret Access Key

String

the secret access key for the stream

Session Token

String

If you are using a session token (see GetSessionToken), specify it here. See also Temporary Security Credentials.

Stream Name

String

The existing Kinesis stream to write to (will not be created if it does not exist). If more than one Kinesis Writer writes to the same stream, recovery is not supported (see Recovering applications. Recovery is based on the stream name at the time the application is first started, so if the stream name changes, to support recovery the application must be dropped and recreated.Recovering applications

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.Supported writer-formatter combinations

Kinesis Writer sample application

Example:

CREATE TARGET KinesisTest USING KinesisWriter (
  regionName:'us-east-1',
  streamName:'myStream',
  accesskeyid:'********************',
  secretaccesskey:'****************************************',
  partitionKey: 'merchantId'
)
FORMAT USING JSONFormatter ()
INPUT FROM PosSource_TransformedStream;

Kinesis Writer monitoring metrics

Metric

Description

Topic-partition Info

The output for each metric is in JSON format, grouped by stream and partition:

  • Previous Batch Info:

    • No. of Records: total count of records produced as part of the last batch

    • Request Size: size of the last record produced to the Kinesis API

    • Time Between Send Calls: time in milliseconds between consecutive send calls to the Kinesis API

  • Total No. of Records Sent: cumulative count of records sent to the Kinesis stream since application start

  • Average Request Size: average size of batches produced to the Kinesis API

  • Avg. Time Between Send Calls: average time in milliseconds between send calls to the Kinesis API

  • Max. Time Between Send Calls: maximum time in milliseconds between send calls to the Kinesis API

  • Total No. of Requests Sent: total number of API requests made to the Kinesis service

  • Average Latency: average time taken to process and send each batch