Skip to main content

GCS Writer

Google Cloud Storage is a cloud storage platform designed to store large, unstructured data sets. It provides unified file storage for data. You can use the GCS Writer to write data to a Google Cloud Storage bucket.

Summary

APIs used/data supported

Google Cloud Storage APIs: Put, Create and List bucket

Supported formatters

Avro, DSV, JSON, Parquet, XML. See Supported writer-formatter combinations.Supported writer-formatter combinations

Supported sources

All sources supported by Striim such as Oracle, SQL Server, MySQL, Amazon S3, or Azure Storage, that generate the following event types: JSONNodeEvent, ParquetEvent, user-defined, WAEvent, XMLNodeEvent. Unsupported sources include those that generate the AvroEvent type.

See Readers overview.Readers overview

Security and authentication

GCS Writer authenticates its connection to Google Cloud Storage using a Google service account key.

GCS Writer supports private endpoints. See Using Private Service Connect with Google Cloud adapters.

Resilience / recovery

  • Supports recovery with at-least-once processing (A1P). See Recovering applications.Recovering applications

  • Any API call to the GCS bucket is retried on a connection failure based on the Connection Retry Policy property (see GCS Writer properties).

Performance

Supports parallel threads to increase throughput to the target. See Creating multiple writer instances (parallel threads).Creating multiple writer instances

Programmability

  • Wizards in the web UI to build pipelines from sources such as databases or apps

  • Striim Flow Designer

  • Striim TQL

Metrics and auditing

Key metrics available through Striim monitoring. See GCS Writer monitoring metrics.

Typical use case and integration

A typical use case is to use Google Cloud Storage as a centralized repository that stores, processes, and secures files of any format. Striim is used to process and store the newly generated event from the source.

GCS Writer overview

Google Cloud Storage is binary large object (blob) storage from Google, and is functionally similar to Amazon S3 or Azure Blob Storage. Similar to Striim's S3 Writer, the GCS Writer stages files locally before they are uploaded to GCS. Like other blob storage, GCS doesn't allow appending to a file; a file is uploaded as is.

The local staging area is located at .striim/<appname>/<bucket-name>/<user-provided-folder-path>/<file>. Once a file is uploaded, it is deleted from the local staging area.

Google supports a number of different methods for uploading to cloud storage (see Uploads and downloads). GCS Writer uses the Upload objects from memory method. Once GCS Writer stages the file locally, it reads the contents at once in a byte array and uploads them to the bucket. This adapter requires you to provide:

  • A Google service account key (which you can download in JSON format from your Google Cloud account).

  • A project ID to which the Google service account has read/write permissions to an existing GCS bucket.