Skip to main content

GCS Writer

Writes to Google Cloud Storage.

GCS Writer properties

property

type

default value

notes

Bucket Name

String

The GCS bucket name. If it does not exist, it will be created (provided Location and Storage Class are specified).

See Setting output names and rollover / upload policies for advanced options.

Note the limitations in Google's Bucket and Object Naming Guidelines. Note particularly that bucket names must be unique not just within your project or account but across all Google Cloud Storage accounts.

See Setting output names and rollover / upload policies for advanced Striim options.

Client Configuration

String

Optionally, specify one or more of the following property-value pairs, separated by commas, to override Google's defaults (see Class RetrySettings):

  • connectionTimeout=<value>: default is 30000

  • msxErrorRetry=<value>: default is 3

  • retryDelay=<value>: default is 30000

Compression Type

String

Set to gzip to compress the output using gzip. Otherwise, leave blank.

Data Encryption Key Passphrase

encrypted password

See Setting encryption policies.

Encryption Policy

String

See Setting encryption policies.

Folder Name

String

Optionally, specify a folder within the specified bucket. If it does not exist, it will be created.

See Setting output names and rollover / upload policies for advanced options.

Location

String

The location of the bucket, which you can find on the bucket's overview tab (see Bucket Locations).

Object Name

String

The base name of the files to be written. See Google's Bucket and Object Naming Guidelines.

See Setting output names and rollover / upload policies for advanced Striim options.

Parallel Threads

Integer

See Creating multiple writer instances.

Partition Key

String

If you enable ParallelThreads, specify a field to be used to partition the events among the threads.  Events will be distributed among multiple folders based on this field's values. 

If the input stream is of any type except WAEvent, specify the name of one of its fields.

If the input stream is of the WAEvent type, specify a field in the METADATA map (see WAEvent contents for change data) using the syntax @METADATA(<field name>), or a field in the USERDATA map (see Adding user-defined data to WAEvent streams), using the syntax @USERDATA(<field name>). If appropriate, you may concatenate multiple METADATA and/or USERDATA fields.WAEvent contents for change data

Private Service Connect Endpoint

String

If using Private Service Connect with Google Cloud Storage, specify the endpoint created in the target Virtual Private Cloud, as described in Private Service Connect support for Google cloud adapters.

Project Id

String

The Google Cloud Platform project for the bucket.

Roll Over on DDL

Boolean

True

Has effect only when the input stream is the output stream of a CDC reader source. With the default value of True, rolls over to a new file when a DDL event is received. Set to False to keep writing to the same file.

Service Account Key

String

The path (from root or the Striim program directory) and file name to the .json credentials file downloaded from Google (see Service Accounts). This file must be copied to the same location on each Striim server that will run this adapter, or to a network location accessible by all servers. The associated service account must have the Storage Legacy Bucket Writer role for the specified bucket.

Storage Class

String

The storage class of the bucket, which you can find on the bucket's overview tab (see Bucket Locations).

Upload Policy

String

eventcount:10000, interval:5m

The upload policy may include eventcount, interval, and/or filesize (see Setting output names and rollover / upload policies for syntax). Cached data is written to GCS every time any of the specified values is exceeded. With the default value, data will be written every five minutes or sooner if the cache contains 10,000 events. When the app is undeployed, all remaining data is written to GCS.

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.Supported writer-formatter combinations

GCS Writer sample application

CREATE APPLICATION testGCS;

CREATE SOURCE PosSource USING FileReader ( 
  wildcard: 'PosDataPreview.csv',
  directory: 'Samples/PosApp/appData',
  positionByEOF:false )
PARSE USING DSVParser (
  header:Yes,
  trimquote:false ) 
OUTPUT TO PosSource_Stream;

CREATE CQ PosSource_Stream_CQ 
INSERT INTO PosSource_TransformedStream
SELECT TO_STRING(data[1]) AS MerchantId,
  TO_DATE(data[4]) AS DateTime,
  TO_DOUBLE(data[7]) AS AuthAmount,
  TO_STRING(data[9]) AS Zip
FROM PosSource_Stream;

CREATE TARGET GCSOut USING GCSWriter  ( 
  bucketName: 'mybucket',
  objectName: 'myobjectname',
  serviceAccountKey: 'conf/myproject-ec6f8b0e3afe.json',
  projectId: 'myproject' ) 
FORMAT USING DSVFormatter  () 
INPUT FROM PosSource_TransformedStream;

END APPLICATION testGCS;