Databricks Writer operational considerations

Use a connection profile for authentication. A connection profile lets you update credentials, including rotating a Service Principal's Client Secret, without editing every application that uses them, and without stopping and restarting each application individually the way inline credentials require.

Retrieve secrets from a vault rather than embedding them inline. Client Secret, Client ID, GCS Service Account Key, and Storage Credential Name can all be sourced from an external vault rather than specified as plain values in your application.

Partition your target tables when using Merge mode. Since Delta Lake tables don't have primary keys, Merge mode relies on scanning to match rows; partitioning significantly reduces the need for full-table scans and improves performance. You cannot partition an existing table after the fact, so decide on partitioning before creating target tables.

Avoid Databricks File System (DBFS) as a staging area for production or sensitive data, on any cloud. Use S3, ADLS Gen2, or GCS instead, as appropriate to your cloud platform.

Do not configure new personal staging locations. This feature has been deprecated by Databricks (and, for Azure, by Microsoft) — use S3, ADLS Gen2, or GCS staging instead.

Use a Connection Profile for GCS staging as well as authentication. A GCS Connection Profile lets you rotate a GCS service account key without editing or restarting every application that uses it.

Place your GCS bucket in the same region as your Databricks workspace. This avoids egress charges. A bucket in a different region will still work, but at additional cost.

Databricks Writer halts the application, rather than silently failing or dropping data, when authentication or permissions are lost while an application is running. This is verified behavior for each of the following scenarios:

In each case, restoring the missing permission or credential and restarting the application resolves the halt.

You can change certain configuration by undeploying the application, altering the relevant setting, testing the connection, and redeploying:

Multiple Striim applications can use the same Service Principal credentials simultaneously without authentication conflicts. This is useful when several pipelines write to the same Databricks workspace under one service identity.

Service Principal authentication and GCS staging work together end-to-end, for both initial load and CDC — data flows from your source, through GCS staging, into Databricks via COPY INTO, authenticated throughout by the Service Principal.

If the GCS bucket specified in gcsBucketName doesn't exist, Databricks Writer attempts to create it automatically, provided the GCS service account has the storage.buckets.create permission. If that permission is missing and the bucket doesn't exist, Databricks Writer reports a permission error rather than silently failing.

If you add an IAM Condition to your GCS service account restricting it to a specific bucket (for example, resource.name.startsWith("projects/_/buckets/<bucket-name>")), Databricks Writer can access that bucket normally, but access to any other bucket fails — confirming the restriction works as intended for tightening the scope of the staging service account's permissions.

Service Principal (M2M OAuth) authentication has been verified against Databricks deployments on Azure, AWS, and Google Cloud, confirming it behaves consistently regardless of cloud platform — as expected, since Service Principal authentication is not tied to any cloud-specific mechanism.

Databricks Writer's pre-existing capabilities — write mode behavior (Append Only, Merge, Optimized Merge), key-column and column-map handling, table mapping (including wildcards and multi-table mappings), partitioned-table writes, application recovery after restart, and Ignorable Exception Code handling — have been re-verified against Service Principal authentication and continue to work as previously documented.