Programmer's reference for Firestore Writer

Authentication Type

Enum

SCRAM

The mechanism used to authenticate to the Firestore database. Valid values are SCRAM (username and password), Service Account Key, and Application Default. See Initial setup for Firestore Writer for how to set up each mechanism.

The Application Default mechanism supports authentication in two scenarios: a Google Cloud VM with a service account attached that has the appropriate permissions, and a user logged in with gcloud auth application-default login. It does not support a service account key supplied through the GOOGLE_APPLICATION_CREDENTIALS environment variable, or service account impersonation using gcloud auth application-default login --impersonate-service-account. This is a limitation of Firestore when connecting through the MongoDB driver.

Connection URL

String

Copy the connection string from your Google Cloud Firestore console (Connect > SCRAM) and use it here. Do not modify or replace any placeholders in the connection string; Striim handles the credentials securely when establishing the connection. Use the SCRAM-tab connection string for all authentication types, including Service Account Key and Application Default.

The connection string is in MongoDB URI format, for example mongodb://<username>:<password>@abc.asia-south1.firestore.goog:443/striim-mongo-firestore-test?loadBalanced=true&tls=true&authMechanism=SCRAM-SHA-256&retryWrites=false. When the connection string includes authMechanism=SCRAM-SHA-256, the MongoDB driver requires the username and password to be present, either in the connection string or in the Username and Password fields.

Username

String

The username of the Firestore database user, used when Authentication Type is SCRAM. You may supply the username here or as part of the Connection URL.

Password

Encrypted password

The password of the Firestore database user, used when Authentication Type is SCRAM. You may supply the password here or as part of the Connection URL.

Service Account Key

String

The path and file name of the .json service account key file downloaded from Google, for example /home/striim/key.json. Appears, and is required, only when Authentication Type is Service Account Key.

Batch Policy

String

EventCount:1000, Interval:30

This property defines how much data Striim can accumulate and how much time can pass before Striim sends a batch of data to Firestore. With the default setting, data will be written at least every 30 seconds or when the buffer accumulates 1,000 events.

Checkpoint Collection

String

Specify a collection (in the format <database>.<collection>) in which Striim will store the target checkpoint required for exactly-once processing (E1P). When this property is not specified, recovery provides at-least-once processing (A1P), which means that after recovery the target may contain some duplicate events.

Collections

String

Specify the source-target collection mapping, including any wildcards or key columns, in Striim's standard format, for example src.emp,firestore-db.emp. The target collections must exist in Firestore before the application is started: Firestore Writer does not create collections automatically.

Connection Profile Name

String

The name of the Firestore connection profile, in the format <namespace>.<profile name>, for example admin.FirestoreCP. Required. See Firestore connection profile properties.

Connection Retry Policy

String

retryInterval=60, maxRetries=3

The retry policy in case of a connection exception: retryInterval is the number of seconds between attempts and maxRetries is the number of attempts.

Excluded Collections

String

When a wildcard is specified for Collections, you may specify here any collections you wish to exclude, in the same format as Collections, for example firestore-db.exclude.

Ignorable Exception Code

String

Exceptions to ignore during data write, for example NO_OP_EXCEPTION. Note that NO_OP_EXCEPTION has no effect for no-op deletes, because Firestore acknowledges no-op deletes rather than raising an exception (see Operational considerations for Firestore Writer).

Key Separator

String

:

The delimiter Striim uses when constructing the document key from multiple fields. If the key field values may contain :, change this to a sequence of characters that will not appear in the values.

Ordered Writes

Boolean

True

Controls whether bulk writes are ordered. With the default value of True, events within a batch are applied in order.

Overload Retry Policy

String

retryInterval=10, maxRetries=3

Determines the interval in seconds and the number of attempts the adapter retries when Firestore returns an Aborted (112) error, which indicates the database is overloaded or the operation conflicted with another. See Understand error codes > Aborted (112) in Google's documentation for more details.

Parallel Threads

Integer

Enables target parallelism for initial load only.

Upsert Mode

Boolean

False

Set to True to process inserts and updates as upserts.