Kinesis Writer
Writes to an Amazon Kinesis stream.
Limitations:
Only sync mode is supported.
Parallel threads are not supported.
Do not write to a target stream with more than 249 shards.
Do not merge, split, start or stop encryption on, or change the shard count of the target stream.
QUIESCE is not supported in this release.
If multiple writers write to the same shard, there may be duplicate events after Recovering applications (in other words, exactly-once processing cannot be guaranteed).
The maximum record length cannot exceed 1MB, including the metadata Striim adds to support recovery. (This is a limitation in Kinesis, not Striim.)
Kinesis Writer properties
property | type | default value | notes |
|---|---|---|---|
Access Key ID | String | Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with write permission on the stream. When Striim is running in Amazon EC2 and there is an IAM role with that permission associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role. | |
Batch Policy | String | Size:900000, Interval:1 | The batch policy includes size and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to With the default setting, data will be written every second or sooner if the an event pushes the buffer past 900,000 bytes. The buffer will expand as necessary to include that last event in the batch. |
Mode | String | Sync | Do not change from default. |
Partition Key | String | Optionally, specify a field to be used to partition the events among multiple shards. See Kafka Writer for more details. | |
Region Name | String | the AWS region of the stream (see AWS Regions and Endpoints) | |
Secret Access Key | String | the secret access key for the stream | |
Session Token | String | If you are using a session token (see GetSessionToken), specify it here. See also Temporary Security Credentials. | |
Stream Name | String | The existing Kinesis stream to write to (will not be created if it does not exist). If more than one Kinesis Writer writes to the same stream, recovery is not supported (see Recovering applications. |
This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.
Kinesis Writer sample application
Example:
CREATE TARGET KinesisTest USING KinesisWriter ( regionName:'us-east-1', streamName:'myStream', accesskeyid:'********************', secretaccesskey:'****************************************', partitionKey: 'merchantId' ) FORMAT USING JSONFormatter () INPUT FROM PosSource_TransformedStream;