S3 Reader
Reads from Amazon S3.
See Supported reader-parser combinations) for parsing options.
S3 Reader properties
property | type | default value | notes |
---|---|---|---|
accesskeyid | String | Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with read permissions (ListBucket, GetObject) on the bucket. When Striim is running in Amazon EC2 and there is an IAM role with those permissions associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role. | |
blocksize | Integer | 64 | amount of data in KB for each read operation |
bucketname | String | S3 bucket to read from | |
clientconfiguration | String | Optionally, specify one or more of the following property-value pairs, separated by commas. If you access S3 through a proxy server, specify it here using the syntax Specify any of the following to override Amazon's defaults:
See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html for more information about these settings. | |
compressiontype | String | Set to | |
foldername | String | Specify a folder within the bucket, or leave blank to read from the root. | |
objectnameprefix | String | The start of the names of the files to be read. For example, | |
secretaccesskey | encrypted password | Specify the AWS secret access key for the specified access key. |
The output type is WAevent except when using Avro Parser or JSONParser.
S3 Reader example
CREATE SOURCE S3Source USING S3Reader ( bucketname:'MyBucket', objectnameprefix:'posdata', accesskeyid:'********************', secretaccesskey:'****************************************', foldername:'MyFolder' ) PARSE USING DSVParser () OUTPUT TO S3SourceStream;
Create an IAM user for use with S3 Reader or S3 Writer
Note
The user interfaces described below are subject to change by Amazon at any time.
Follow these steps to create an IAM user with the necessary permissions to read from or write to an S3 bucket and to get the access key and secret access key for that user. If appropriate in your environment you may use the same IAM user for both S3 Reader and S3 Writer.
If the bucket does not already exist, create it.
Select the bucket and click Copy ARN.
Go to the AWS Policy Generator at https://awspolicygen.s3.amazonaws.com/policygen.html
For Select Type of Policy, select IAM Policy.
For AWS Service, select Amazon S3.
Select the individual actions you want to allow or select All Actions.
In the Amazon Resource Name (ARN) field, paste the bucket's ARN that you copied.
Click Add Statement. You should see something similar to this:
Click Generate Policy.
Copy the Policy JSON Document and close the dialog.
Go to the IAM Policies page and click Create policy.
Select the JSON tab, replace the existing text with the policy JSON document you copied, and click Next: Tags > Next: Review.
Enter a descriptive name for the policy (make a note of this as you will need it later), optionally enter a description, and click Create Policy.
Go to the IAM Users page and click Add users.
Enter a name for the IAM user, select Access key, click Next: Permissions.
Select Attach existing policies directly, in Filter policies enter the name of the policy you created, select the policy, and click Next: Tags > Next: Review > Create user.
Click Download .csv. This file contains the access key and secret access key you must provide to S3 Reader and/or S3 Writer.