Skip to main content

Kinesis Writer

Writes to an Amazon Kinesis stream.

Limitations:

  • Only sync mode is supported.

  • Parallel threads are not supported.

  • Do not write to a target stream with more than 249 shards.

  • Do not merge, split, start or stop encryption on, or change the shard count of the target stream.

  • QUIESCE is not supported in this release.

  • If multiple writers write to the same shard, there may be duplicate events after Recovering applications (in other words, exactly-once processing cannot be guaranteed).

  • The maximum record length cannot exceed 1MB, including the metadata Striim adds to support recovery. (This is a limitation in Kinesis, not Striim.)

Kinesis Writer properties

property

type

default value

notes

Access Key ID

String

Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with write permission on the stream.

When Striim is running in Amazon EC2 and there is an IAM role with that permission associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role.

Batch Policy

String

Size:900000, Interval:1

The batch policy includes size and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to Size:-1.

With the default setting, data will be written every second or sooner if the an event pushes the buffer past 900,000 bytes. The buffer will expand as necessary to include that last event in the batch.

Mode

String

Sync

Do not change from default.

Partition Key

String

Optionally, specify a field to be used to partition the events among multiple shards. See Kafka Writer for more details.

Region Name

String

the AWS region of the stream (see AWS Regions and Endpoints)

Secret Access Key

String

the secret access key for the stream

Session Token

String

If you are using a session token (see GetSessionToken), specify it here. See also Temporary Security Credentials.

Stream Name

String

The existing Kinesis stream to write to (will not be created if it does not exist). If more than one Kinesis Writer writes to the same stream, recovery is not supported (see Recovering applications.Recovering applications

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.Supported writer-formatter combinations

Kinesis Writer sample application

Example:

CREATE TARGET KinesisTest USING KinesisWriter (
  regionName:'us-east-1',
  streamName:'myStream',
  accesskeyid:'********************',
  secretaccesskey:'****************************************',
  partitionKey: 'merchantId'
)
FORMAT USING JSONFormatter ()
INPUT FROM PosSource_TransformedStream;