Delta Lake Writer properties
When creating a Delta Lake Writer target in TQL, you must specify values for the Connection URL, Hostname, Personal Access Token, and Tables properties. If not specified, the other properties will use their default values.
property | type | default value | notes |
---|---|---|---|
CDDL Action | String | Process | |
Connection Retry Policy | String | initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=1m, maxAttempts=5, totalTimeout=10m | Do not change unless instructed to by Striim support. |
Connection URL | String | the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster) | |
External Stage Type | enum |
| With the default value (not recommended), events are staged to DBFS storage at the path specified in Stage Location. If running Databricks on AWS, set to If running Azure Databricks, set to |
Hostname | String | the Server Hostname from the JDBC/ODBC tab of the Databricks cluster's Advanced options (see Get connection details for a cluster) | |
Ignorable Exception Code | String | Set to TABLE_NOT_FOUND to prevent the application from crashing when Striim tries to write to a table that does not exist in the target. See Handling "table not found" errors for more information. | |
Mode | enum | AppendOnly | With the default value AppendOnly:
Set to Merge to handle updates and deletes as updates and deletes instead. In Merge mode:
|
Optimized Merge | Boolean | false | In Flow Designer, this property will be displayed only when Mode is Merge. Set to True only when Mode is MERGE and the target's input stream is the output of an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events will include partial records. For example, with Oracle Reader, when supplemental logging has not been enabled for all columns, partial records are sent for updates. When the source events will always include full records, leave this set to false. |
Parallel Threads | Integer | Not supported when Mode is Merge. | |
Personal Access Token | encrypted password | Used to authenticate with the Databricks cluster (see Generate a personal access token). The user associated with the token must have read and write access to DBFS (see Important information about DBFS permissions). If table access control has been enabled, the user must also have MODIFY and READ_METADATA (see Data object privileges - Data governance model). | |
Stage Location | String |
| When the External Stage Type is DBFSROOT, the path to the staging area in DBFS, for example, |
Tables | String | The name(s) of the table(s) to write to. The table(s) must exist in the database. Specify target table names in uppercase as When the target's input stream is a user-defined event, specify a single table. The only special character allowed in target table names is underscore ( When the input stream of the writer is the output of a DatabaseReader, IncrementalBatchReader, or SQL CDC source it can write to multiple tables. In this case, specify the names of both the source and target tables. You may use the % wildcard for the table names, but not for the schema or database. If the reader uses three-part names, you must use them here as well. SQL Server source table names must be specified in three parts when the source is Database Reader or Incremental Batch Reader ( source.emp,target_database.emp source_schema.%,target_database.% source_database.source_schema.%,target_database.% MySQL and Oracle names are case-sensitive, SQL Server names are not. Specify names as See Mapping columns for additional options. | |
Upload Policy | String | eventcount:10000, interval:60s | The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Buffered data is written to the storage account every time any of the specified values is exceeded. With the default value, data will be written every 60 seconds or sooner if the buffer contains 10,000 events. When the app is undeployed, all data remaining in the buffer is written to the storage account. |
Azure Data Lake Storage (ADLS) Gen2 properties for Delta Lake Writer
property | type | default value | notes |
---|---|---|---|
Azure Account Access Key | encrypted password | the account access key from Storage accounts > <account name> > Access keys | |
Azure Account Name | String | the name of the Azure storage account for the blob container | |
Azure Container Name | String | the blob container name from Storage accounts > <account name> > Containers If it does not exist, it will be created. |
S3 properties for Delta Lake Writer
property | type | default value | notes |
---|---|---|---|
S3 Access Key | String | an AWS access key ID (created on the AWS Security Credentials page) for a user with read and write permissions on the bucket If the Striim host has default credentials stored in the | |
S3 Bucket Name | String | Specify the S3 bucket to be used for staging. If it does not exist, it will be created. | |
S3 Region | String | the AWS region of the bucket | |
S3 Secret Access Key | encrypted password | the secret access key for the access key If the Striim host has default credentials stored in the |