Databricks Writer programmer's reference

Authentication Type

enum

PersonalAccessToken

Appears in Flow Designer only when Connection Profile is False.

The simplest way to use Microsoft Entra ID (OAuth) is to create a connection profile (see Connection profiles). Alternatively, select Manual OAuth and follow the instructions in Configuring Microsoft Entra ID (formerly Azure Active Directory) for Databricks Writer manually. In that case, specify Client ID, Client Secret, Refresh Token, and Tenant ID.

With the default setting PersonalAccessToken, Striim's connection to Databricks is authenticated using the token specified in Personal Access Token.

CDDL Action

enum

Process

See Handling schema evolution.

If TRUNCATE commands may be entered in the source and you do not want to delete events in the target, precede the writer with a CQ with the select statement SELECT * FROM <input stream name> WHERE META(x, OperationName).toString() != 'Truncate'; (replacing <input stream name> with the name of the writer's input stream). Note that there will be no record in the target that the affected events were deleted.

Client ID

string

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Manual OAuth.

This property is required when Manual OAuth is selected as the value of the Authentication Type property.

Client Secret

encrypted password

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Manual OAuth.

This property is required when Manual OAuth is selected as the value of the Authentication Type property.

Connection Profile Name

Enum

Appears in Flow Designer only when Use Connection Profile is True. See Connection profiles.

Connection Retry Policy

String

initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=1m, maxAttempts=5, totalTimeout=10m

Do not change unless instructed to by Striim support.

Connection URL

String

Appears in Flow Designer only when Connection Profile is False.

Provide the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster). If the URL starts with jdbc:spark:// change that to jdbc:databricks:// (this is required by the upgraded driver bundled with Striim).

External Stage Connection Profile Name

enum

Appears in Flow Designer only when Use Connection Profile is True and External Stage Type is ADLSGen2 or S3.

Select or specify the name of the connection profile for the external stage. (When Databricks Writer uses a connection profile, you must use a connection profile for ADLSGen2 or S3 as well.)

External Stage Type

enum

DBFSROOT

Set to ADLSGen2 or S3 to match the stage type you chose in Choose which staging area to use.

Ignorable Exception Code

String

Set to TABLE_NOT_FOUND to prevent the application from terminating when Striim tries to write to a table that does not exist in the target. See Handling "table not found" errors for more information.

Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE).

Mode

enum

AppendOnly

Set to Merge if that was your choice in Choose which writing mode to use.

Optimized Merge

Boolean

false

Appears in Flow Designer only when Mode is Merge.

Set to True only when Mode is MERGE and the target's input stream is the output of an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events will include partial records. For example, with Oracle Reader, when supplemental logging has not been enabled for all columns, partial records are sent for updates. When the source events will always include full records, leave this set to False.

When Optimized Merge is True, each primary key update is handled in a separate write operation. If the source has frequent primary key updates, this may lead to a decline in write performance compared with Optimized Merge = False.

Parallel Threads

Integer

Not supported when Mode is Merge.

See Creating multiple writer instances (parallel threads).

Personal Access Token

encrypted password

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Personal Access Token.

Used to authenticate with the Databricks cluster (see Generate a personal access token). The user associated with the token must have read and write access to DBFS (see Important information about DBFS permissions). If table access control has been enabled, the user must also have MODIFY and READ_METADATA (see Data object privileges - Data governance model).

Personal Staging User Name

String

Personal staging locations have been deprecated by AWS (see Create metastore-level storage) and Microsoft (see Create metastore-level storage).

Refresh Token

encrypted password

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Manual OAuth.

This property is required when Manual OAuth is selected as the value of the Authentication Type property. The token expires in 90 days, after which the application will halt. To avoid that, use a connection profile (see Connection profiles), which will allow you to update the token without stopping the application. Alternatively, prior to expiry stop the application and update the token.

Stage Location

String

/

See  Choose which staging area to use.

Tables

String

The name(s) of the table(s) to write to. If not using a Database Reader source with Create Schema enabled or a wizard with initial schema creation, the table(s) must already exist in the database.

Specify target table names as <catalog>.<database>.<table>. Specifying the catalog is required for initial schema creation with Unity Catalog and not specifying it may result in errors if a table in another catalog has the same name. Names are case-insensitive.

When the target's input stream is a user-defined event, specify a single table.

The only special character allowed in target table names is underscore (_).

When the input stream of the target is the output of a DatabaseReader, IncrementalBatchReader, or SQL CDC source (that is, when replicating data from one database to another), it can write to multiple tables. In this case, specify the names of both the source and target tables. You may use the % wildcard only for tables, not for schemas or databases. If the reader uses three-part names, you must use them here as well. Note that Oracle CDB/PDB source table names must be specified in two parts when the source is Database Reader or Incremental Batch reader (schema.%,schema.%) but in three parts when the source is Oracle Reader or OJet ((database.schema.%,schema.%). Note that SQL Server source table names must be specified in three parts when the source is Database Reader or Incremental Batch Reader (database.schema.%,schema.%) but in two parts when the source is MS SQL Reader or MS Jet (schema.%,schema.%). Examples:

source.emp,target_database.emp
source_schema.%,target_catalog.target_database.%
source_database.source_schema.%,target_database.%
source_database.source_schema.%,
  target_catalog.target_database.%

MySQL and Oracle names are case-sensitive, SQL Server names are not. Specify names as <schema name>.<table name> for MySQL and Oracle and as <database name>.<schema name>.<table name> for SQL Server.

If a column in the target table is not mapped to a column in the source (see When is a target column not mapped to a source column?):

See Mapping columns and Defining relations between source and target using ColumnMap and KeyColumns for additional options.

Tenant ID

String

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Manual OAuth.

This property is required when Manual OAuth is selected as the value of the Authentication Type property.

Upload Policy

String

eventcount:100000, interval:60s

The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Buffered data is written to the external stage every time any of the specified values is exceeded. With the default value, data will be written every 60 seconds or sooner if the buffer contains 100,000 events. When the app is quiesced, any data remaining in the buffer is written to the storage account; when the app is undeployed, any data remaining in the buffer is discarded.

Use Connection Profile

Boolean

False

Set to True to use a connection profile instead of specifying the connection properties here.

Azure Account Access Key

encrypted password

When Authentication Type is set to ServiceAccountKey, specify the account access key from Storage accounts > <account name> > Access keys.

When Authentication Type is set to AzureAD, this property is ignored in TQL and not displayed in the Flow Designer.

Azure Account Name

String

the name of the Azure storage account for the blob container

Azure Container Name

String

striim-deltalakewriter-container

the blob container name from Storage accounts > <account name> > Containers

If it does not exist, it will be created.

S3 Access Key

String

an AWS access key ID (created on the AWS Security Credentials page) for a user with read and write permissions on the bucket

If the Striim host has default credentials stored in the .aws directory, you may leave this blank.

S3 Bucket Name

String

striim-deltalake-bucket

Specify the S3 bucket to be used for staging. If it does not exist, it will be created.

S3 Region

String

us-west-1

the AWS region of the bucket

S3 Secret Access Key

encrypted password

the secret access key for the access key

If the Striim host has default credentials stored in the .aws directory, you may leave this blank.

Authentication Type

enum

PersonalAccessToken

Appears in Flow Designer only when Connection Profile is False.

With the default setting PersonalAccessToken, Striim's connection to Databricks is authenticated using the token specified in Personal Access Token.

Set to EntraID to authenticate using Microsoft Entra (formerly Azure Active Directory). In this case, specify Client ID, Client Secret, Refresh Token, and Tenant ID. See Databricks authentication mechanisms for details.Databricks authentication mechanisms

CDDL Action

enum

Process

See Handling schema evolution.

If TRUNCATE commands may be entered in the source and you do not want to delete events in the target, precede the writer with a CQ with the select statement ELECT * FROM <input stream name> WHERE META(x, OperationName).toString() != 'Truncate'; (replacing <input stream name> with the name of the writer's input stream). Note that there will be no record in the target that the affected events were deleted.

Client ID

string

This property is required when AzureAD authentication is selected as the value of the Authentication Type property.

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Entra ID.

Client Secret

encrypted password

This property is required when AzureAD authentication is selected as the value of the Authentication Type property.

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Entra ID.

Connection Retry Policy

String

initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=1m, maxAttempts=5, totalTimeout=10m

Do not change unless instructed to by Striim support.

Connection URL

String

Provide the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster). If the URL starts with jdbc:spark:// change that to jdbc:databricks:// (this is required by the upgraded driver bundled with Striim).

Appears in Flow Designer only when Connection Profile is False.

External Stage Type

enum

DBFSROOT

With the default value (not recommended), events are staged to DBFS storage at the path specified in Stage Location. To use an external stage, your Databricks instance should be using Databricks Runtime 11.0 or later. When using serverless compute, you must select PersonalStagingLocation as the stage type.

If running Databricks on AWS and you want to use S3, set to S3 and set the S3 properties as detailed below.

In Striim 4.2.0.1 or later only, if running Databricks on AWS and you want to use a Personal Staging Area, see Databricks Writer properties.Databricks Writer properties

If running Azure Databricks and you want to use a personal staging location, see Databricks Writer properties. TKK this has been deprecated, see https://webaction.atlassian.net/browse/DEV-38870Databricks Writer properties

If running Azure Databricks and you want to use Azure Data Lake Storage Gen2, set to ADLSGen2 and set the ADLS properties as detailed below.

Hostname

String

the Server Hostname from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster)

Ignorable Exception Code

String

Set to TABLE_NOT_FOUND to prevent the application from terminating when Striim tries to write to a table that does not exist in the target. See Handling "table not found" errors for more information.

Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE).

Mode

enum

AppendOnly

With the default value AppendOnly:

Set to Merge to handle updates and deletes as updates and deletes instead. In Merge mode:

Optimized Merge

Boolean

false

Appears in Flow Designer only when Mode is Merge.

Set to True only when Mode is MERGE and the target's input stream is the output of an HP NonStop reader, or Oracle Reader source and the source events will include partial records. For HP NonStop, “partial records” means AUDITCOMPRESS on; and for Oracle it means ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS; (rather than ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;).

Parallel Threads

Integer

Not supported when Mode is Merge.

See Creating multiple writer instances (parallel threads).

Personal Access Token

encrypted password

Used to authenticate with the Databricks cluster (see Generate a personal access token). The user associated with the token must have read and write access to DBFS (see Important information about DBFS permissions). If table access control has been enabled, the user must also have MODIFY and READ_METADATA (see Data object privileges - Data governance model).

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Personal Access Token.

Personal Staging User Name

String

When External Stage Type is PersonalStagingLocation, set as desecribed in Databricks Writer properties or Databricks Writer properties. TKK this has been deprecated, see https://webaction.atlassian.net/browse/DEV-38870Databricks Writer propertiesDatabricks Writer properties

Refresh Token

encrypted password

This property is required when AzureAD authentication is selected as the value of the Authentication Type property.

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Entra ID.

Stage Location

String

/

When the External Stage Type is DBFSROOT, the path to the staging area in DBFS, for example, /StriimStage/.

Tables

String

The name(s) of the table(s) to write to. The table(s) must exist in the database.

Specify target table names in uppercase as <CATALOG>.<DATABASE>.<TABLE>. Not specifying the catalog (<DATABASE>.<TABLE>) may result in errors if a table in another catalog has the same name.

When the target's input stream is a user-defined event, specify a single table.

The only special character allowed in target table names is underscore (_).

When the input stream of the target is the output of a DatabaseReader, IncrementalBatchReader, or SQL CDC source (that is, when replicating data from one database to another), it can write to multiple tables. In this case, specify the names of both the source and target tables. You may use the % wildcard only for tables, not for schemas or databases. If the reader uses three-part names, you must use them here as well. Note that Oracle CDB/PDB source table names must be specified in two parts when the source is Database Reader or Incremental Batch reader (schema.%,schema.%) but in three parts when the source is Oracle Reader or OJet ((database.schema.%,schema.%). Note that SQL Server source table names must be specified in three parts when the source is Database Reader or Incremental Batch Reader (database.schema.%,schema.%) but in two parts when the source is MS SQL Reader or MS Jet (schema.%,schema.%). Examples:

source.emp,target_database.emp
source_schema.%,target_catalog.target_database.%
source_database.source_schema.%,target_database.%
source_database.source_schema.%,target_catalog.target_database.%

MySQL and Oracle names are case-sensitive, SQL Server names are not. Specify names as <schema name>.<table name> for MySQL and Oracle and as <database name>.<schema name>.<table name> for SQL Server.

See Mapping columns for additional options.

Tenant ID

String

This property is required when EntraID authentication is selected as the value of the Authentication Type property.

Appears in Flow Designer only when Connection Profile is False and Authentication Type is Entra ID.

Upload Policy

String

eventcount:100000, interval:60s

The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Buffered data is written to the storage account every time any of the specified values is exceeded. With the default value, data will be written every 60 seconds or sooner if the buffer contains 100,000 events. When the app is quiesced, any data remaining in the buffer is written to the storage account; when the app is undeployed, any data remaining in the buffer is discarded.

Use Connection Profile

Boolean

False

See Connection profiles.