Skip to main content

Striim Cloud 4.1.2 documentation

S3 Reader

Reads from Amazon S3.

See Supported reader-parser combinations) for parsing options.

S3 Reader properties

property

type

default value

notes

accesskeyid

String

Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with read permissions (ListBucket, GetObject) on the bucket.

When Striim is running in Amazon EC2 and there is an IAM role with those permissions associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role.

blocksize

Integer

64

amount of data in KB for each read operation

bucketname

String

S3 bucket to read from

clientconfiguration

String

Optionally, specify one or more of the following property-value pairs, separated by commas.

If you access S3 through a proxy server, specify it here using the syntax ProxyHost=<IP address>,ProxyPort=<port number>,ProxyUserName=<user name>,ProxyPassword=<password>. Omit the user name and password if not required by your proxy server.

Specify any of the following to override Amazon's defaults:

  • ConnectionTimeout=<timeout in milliseconds>: how long to wait to establish the HTTP connection, default is 50000

  • MaxErrorRetry=<number of retries>: the number of times to retry failed requests (for example, 5xx errors), default is 3

  • SocketErrorSizeHints=<size in bytes>: TCP buffer size, default is 2000000

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html for more information about these settings.

compressiontype

String

Set to gzip when the files to be read are in gzip format. Otherwise, leave blank.

foldername

String

Specify a folder within the bucket, or leave blank to read from the root.

objectnameprefix

String

The start of the names of the files to be read. For example, myfile will read myfile*.*. Specify * to read all files.

secretaccesskey

encrypted password

Specify the AWS secret access key for the specified access key.

The output type is WAevent except when using Avro Parser  or JSONParser.

S3 Reader example

CREATE SOURCE S3Source USING S3Reader (
  bucketname:'MyBucket',
  objectnameprefix:'posdata',
  accesskeyid:'********************',
  secretaccesskey:'****************************************',
  foldername:'MyFolder'
)
PARSE USING DSVParser ()
OUTPUT TO S3SourceStream;

Create an IAM user for use with S3 Reader or S3 Writer

Note

The user interfaces described below are subject to change by Amazon at any time.

Follow these steps to create an IAM user with the necessary permissions to read from or write to an S3 bucket and to get the access key and secret access key for that user. If appropriate in your environment you may use the same IAM user for both S3 Reader and S3 Writer.

  1. If the bucket does not already exist, create it.

  2. Select the bucket and click Copy ARN.

    S3IAMCopyARN.png

  3. Go to the AWS Policy Generator at https://awspolicygen.s3.amazonaws.com/policygen.html

  4. For Select Type of Policy, select IAM Policy.

  5. For AWS Service, select Amazon S3.

  6. Select the individual actions you want to allow or select All Actions.

  7. In the Amazon Resource Name (ARN) field, paste the bucket's ARN that you copied.

  8. Click Add Statement. You should see something similar to this:

    S3IAMCreatePolicy.png

  9. Click Generate Policy.

  10. Copy the Policy JSON Document and close the dialog.

    S3IAMPolicyJSONDocument.png

  11. Go to the IAM Policies page and click Create policy.

    S3IAMPoliciesPage.png

  12. Select the JSON tab, replace the existing text with the policy JSON document you copied, and click Next: Tags > Next: Review.

    S3IAMPolicyJSONDocument.png

  13. Enter a descriptive name for the policy (make a note of this as you will need it later), optionally enter a description, and click Create Policy.

    S3IAMReviewPolicy.png

  14. Go to the IAM Users page and click Add users.

    S3IAMUsersPage.png

  15. Enter a name for the IAM user, select Access key, click Next: Permissions.

    S3IAMAddUser1.png

  16. Select Attach existing policies directly, in Filter policies enter the name of the policy you created, select the policy, and click Next: Tags > Next: Review > Create user.

    S3IAMSetPermissions.png

  17. Click Download .csv. This file contains the access key and secret access key you must provide to S3 Reader and/or S3 Writer.

In this section: