Initial setup for Databricks Writer
Before you can use Databricks Writer in a Striim application, you must complete one-time setup: choosing and configuring an authentication method, choosing and configuring a staging area, and granting the required Databricks permissions. Some of this setup is common to all three cloud platforms; the rest is specific to whether your Databricks instance runs on Azure, AWS, or Google Cloud.
This page covers the common setup first, then the setup specific to each cloud endpoint. If you already know which cloud you're using, you can skip directly to that subsection after reading Common Setup.
Setup steps common to all cloud providers
What setup involves
Regardless of which cloud your Databricks instance runs on, setting up Databricks Writer involves three things:
Choose an authentication method and complete its prerequisites (a Personal Access Token, or Service Principal credentials, or, on Azure, Microsoft Entra ID).
Choose a staging area and complete its prerequisites (Databricks Writer stages data in cloud storage before loading it into Delta Lake tables. See Batching and Upload Policy.)
Grant the required Databricks permissions to whichever identity (PAT user, Service Principal, or Entra ID identity) will connect to Databricks.
Databricks permissions required for any authentication method
Whichever authentication method you use, the identity connecting to Databricks needs the following permissions:
Permission | Scope | Purpose |
|---|---|---|
USE CATALOG | Catalog | Use the target catalog |
USE SCHEMA | Schema | Use the target schema |
CREATE TABLE | Schema | Create tables (only needed if Databricks Writer will create tables) |
MODIFY | Table | Perform INSERT, UPDATE, and DELETE operations |
SELECT | Table | Perform read operations during merge |
CREATE SCHEMA | Catalog | Create target schema. This is required when the target schema should be auto-created using an automated IL-CDC wizard. |
Authentication methods available by cloud
Authentication Method | Azure Databricks | Databricks on AWS | Databricks on Google Cloud |
|---|---|---|---|
Manual OAuth | ✓ Available | ✓ Available | ✓ Available |
Microsoft Entra ID (formerly Azure Active Directory) | ✓ Available | Not applicable | Not applicable |
Personal Access Token | ✓ Available | ✓ Available | ✓ Available |
Service Principal (M2M OAuth) | ✓ Available | ✓ Available | ✓ Available |
Personal Access Token, Manual OAuth, and Service Principal (M2M OAuth) work the same way regardless of which cloud your Databricks instance runs on, and are set up once in this Common Setup section. Microsoft Entra ID is specific to Azure Databricks and is described in Setup for Azure Databricks below.
Manual OAuth setup
Manual OAuth is available as an authentication method on all three clouds (authenticationType: 'ManualOAuth'). The only Manual OAuth configuration steps we have grounded detail for are the Entra-ID flavor used on Azure Databricks — see Setup for Azure Databricks below.
Personal access token setup
Create a personal access token for Striim to use to authenticate with the Databricks cluster. This works the same way regardless of which cloud your Databricks instance runs on.
On Azure Databricks: follow Databricks' documentation on generating a personal access token for Azure Databricks automation. The user associated with the token must have read and write access to DBFS. If table access control has been enabled, the user must also have MODIFY and READ_METADATA permissions.
On Databricks on AWS: follow Databricks' documentation on Databricks personal access tokens for users. Grant the user associated with the token View and Create permissions on the Databricks File System. If table access control has been enabled, also grant MODIFY and READ_METADATA privileges.
On Databricks on Google Cloud: follow Databricks' documentation on generating a personal access token for Databricks on Google Cloud. Grant the user associated with the token read and write access to DBFS. If table access control has been enabled, also grant MODIFY and READ_METADATA privileges.
Once you have your token, see Databricks Writer programmer's reference for the exact property (personalAccessToken).
Service Principal (M2M OAuth) setup
Service Principal-based Machine-to-Machine (M2M) OAuth authentication is an alternative to a Personal Access Token. Unlike Microsoft Entra ID (which is Azure-specific), Service Principal authentication works identically regardless of which cloud your Databricks instance runs on, so it's covered once here rather than repeated in each cloud-specific subsection below.
Why use Service Principal authentication: it suits long-running or automated applications, since Striim can refresh the access token automatically rather than relying on a token you manage manually.
Step 1: Create a Service Principal in Databricks
Log in to the Databricks account console as an account admin.
In the sidebar, click User management.
Under Service Principals, create a new Service Principal.
After creation, Databricks displays the Service Principal's details, including a UUID. This UUID is the Client ID you'll use for M2M authentication.
Step 2: Generate a Client Secret
On the Service Principal's details page, go to the Credentials & Secrets tab.
Create a Client Secret.
Note the displayed Client Secret — this is required for M2M authentication and will not be shown again.
Step 3: Grant Workspace Access
In the Databricks account console, go to Workspaces and select the workspace Databricks Writer will use.
Click the Permissions tab.
Add the Service Principal with User permission.
Step 4: Grant Catalog, Schema, and Table Permissions
Grant the Service Principal the permissions listed in Databricks permissions required for any authentication method above (USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY,SELECT, CREATE CATALOG) on the catalog, schema, and tables Databricks Writer will use.
Once you've completed these four steps, you have everything you need to configure Service Principal authentication in Databricks Writer — see Databricks Writer programmer's reference for the exact properties (Authentication Type, Client ID, Client Secret, Connection URL).
Important: Service Principal authentication cannot be combined with DBFS as your staging area, regardless of cloud. Choose S3, ADLS Gen2, or GCS staging instead.
Using a connection profile
A connection profile lets you save authentication (and, on Google Cloud, staging) credentials once and reuse or update them across multiple applications, without editing or restarting each application individually. This works the same way regardless of which cloud your Databricks instance runs on, except for Entra ID, which is Azure-specific (see Setup for Azure Databricks below for Entra ID connection profile setup).
Setting up a Personal Access Token connection profile:
In the Create or Edit Connection Profile dialog, select Databricks as the endpoint type.
Specify the Personal Access Token — the user associated with the token must have read and write access to DBFS, and MODIFY/READ_METADATA if table access control is enabled.
Specify the Connection URL — the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options. If the URL starts with jdbc:spark://, change it to jdbc:databricks://.
When you update the token on a connection profile, applications using that profile automatically switch to the new token when the old one expires.
Setting up a Service Principal connection profile:
In the Create or Edit Connection Profile dialog, select Databricks as the endpoint type.
Specify the Client ID, Client Secret, and Connection URL from your Service Principal setup above.
Service Principal connection profiles work the same way regardless of which cloud your Databricks instance runs on.
Setup for Azure Databricks
Authentication methods for Azure Databricks
Azure Databricks supports:
Manual OAuth — see Manual OAuth setup above (Azure is the only cloud we have grounded Manual OAuth steps for, described below as Microsoft Entra ID configuration)
Microsoft Entra ID (formerly Azure Active Directory) — Azure-specific; described in detail below
Personal Access Token — see Personal access token setup above
Service Principal (M2M OAuth) — see Service Principal (M2M OAuth) setup above
Microsoft Entra ID authentication
Microsoft Entra ID (formerly Azure Active Directory) authentication is specific to Azure Databricks.
We recommend using a connection profile to authenticate with Entra ID rather than configuring it manually, since a connection profile is significantly simpler to set up.
Setting up an Entra ID Connection Profile:
In the Create or Edit Connection Profile dialog, select Databricks as the endpoint type.
Click Sign in using Entra ID and log in with an Entra ID account that has: USE CATALOG and CREATE SCHEMA on the Databricks catalog that will contain the target schema; CAN USE permission on the Databricks workspace; and CAN ATTACH TO on the workspace's compute.
Once you've logged in successfully, close the browser window (if you don't, sign-in will fail) and return to the connection profile page.
Specify the Connection URL — the JDBC URL from the JDBC/ODBC tab of the Databricks cluster's Advanced options. If the URL starts with jdbc:spark://, change it to jdbc:databricks://.
Test the connection.
If you need to configure Entra ID manually (Manual OAuth) instead, in summary this requires: registering the Striim app with the Entra ID identity provider; noting the registered app's Client ID, Client Secret, and Tenant ID; and obtaining an access token and refresh token via the OAuth authorization code flow (either using curl and a browser, or a tool like Postman). The refresh token expires in 90 days; using a connection profile instead of Manual OAuth lets you update the token without stopping the application.
Choose a staging area for Azure Databricks
Writing to Databricks happens in two steps: Striim first writes data to temporary files in a staging area in the cloud, and Databricks then copies the data from those files into its tables. We strongly recommend using Azure Data Lake Storage (ADLS) Gen2 as your staging area. To use an external stage, your Databricks instance must run Databricks Runtime 10.4 or later (10.11 or later if you're using an ADLS Gen2 container specifically). The native Databricks File System (DBFS) should not be used as the staging area for production or sensitive data.
Set up ADLS Gen2 as the staging location:
Create a general-purpose v2 storage account to use with Azure Data Lake Storage Gen2.
Create a service principal and client secret for the storage account (this is a storage-access service principal, distinct from the Service Principal used for Databricks authentication described above).
You may manually create the required container, or let Databricks Writer create it automatically — the default name is striim-deltalakewriter-container.
Note
Personal staging locations have been deprecated by both Databricks and Microsoft. Do not set up a new personal staging location for Azure Databricks.
Setup for Databricks on AWS
Authentication methods for Databricks on AWS
Databricks on AWS supports:
Manual OAuth — see Manual OAuth setup above
Personal Access Token — see Personal access token setup above
Service Principal (M2M OAuth) — see Service Principal (M2M OAuth) setup above
(Microsoft Entra ID is Azure-specific and not applicable on AWS.)
Choose a staging area for Databricks on AWS
Writing to Databricks happens in two steps: Striim first writes data to temporary files in a staging area in the cloud, and Databricks then copies the data from those files into its tables. We strongly recommend using Amazon S3 for staging. The native Databricks File System (DBFS) should not be used as the staging area for production or sensitive data.
Create an S3 bucket to use as a staging location:
If the bucket doesn't already exist, create it.
Select the bucket and copy its ARN.
Go to the AWS Policy Generator. Select IAM Policy as the policy type and Amazon S3 as the AWS service.
Select the actions you want to allow (or select all actions), paste the bucket's ARN into the Amazon Resource Name field, and click Add Statement, then Generate Policy.
Copy the generated policy JSON document.
In IAM, go to Policies and click Create policy. Select the JSON tab, paste in the policy document, and proceed through the wizard, giving the policy a descriptive name you'll remember.
In IAM, go to Users and click Add users. Create a user with an access key, then attach the policy you just created.
Download the resulting .csv file — it contains the access key and secret access key you'll specify in Databricks Writer's properties.
Note
personal staging locations have been deprecated by AWS. Do not set up a new personal staging location for Databricks on AWS.
Setup for Databricks on Google Cloud
Authentication methods for Databricks on Google Cloud
Databricks on Google Cloud supports:
Manual OAuth — see Manual OAuth setup above
Personal Access Token — see Personal access token setup above
Service Principal (M2M OAuth) — see Service Principal (M2M OAuth) setup
(Microsoft Entra ID is Azure-specific and not applicable on Google Cloud.)
Important: Service Principal authentication cannot be used together with DBFS as the staging area, on any cloud including Google Cloud. If you plan to use Service Principal authentication, choose GCS (or another supported staging area) instead of DBFS. See Limitations.
GCS staging setup
Databricks supports direct authentication when loading data from Azure and AWS storage, but not from GCS. Instead, GCS staging requires a Storage Credential of GCP Service Account type, created in Unity Catalog, which is used to read from GCS.
GCS staging can only be used when writing to a Unity Catalog target — it is not available when writing to the legacy Hive metastore.
Setting this up involves two systems and two separate permission grants: you create the Storage Credential and grant it access to your GCS bucket in Google Cloud IAM, and you separately grant a Databricks user or Service Principal permission to use that Storage Credential.
The following steps enable the Databricks cluster to access the GCS bucket to access the stage files.
Enable the Databricks cluster to access the GCS bucket
The following steps enable the Databricks cluster to access the GCS bucket to access the stage files.
Step 1: Create a Storage Credential in Unity Catalog
In your Databricks workspace, go to Catalog > Manage > Credentials (grouped under External Data).
Click Create Credential, select Storage Credential, and set Credential Type to GCP Service Account.
Provide a credential name — you'll use this name later as the storageCredentialName property in Databricks Writer.
Click Create Credential. Databricks creates a Google Cloud service account behind the scenes and displays its details. Copy the service account information — you'll need it in the next step.
Step 2: Grant the Storage Credential's Service Account Access to Your GCS Bucket
In Google Cloud IAM, grant the service account (copied in Step 1) the Storage Object Viewer role on the bucket you plan to use for staging.
To restrict the service account's access to only the specific bucket you intend to use, you can add an IAM Condition based on the resource name (for example, restricting to projects/_/buckets/<your-bucket-name>).
Step 3: Grant READ FILES Permission in Databricks
In your Databricks workspace, go to Catalog > Manage > Credentials and open the Storage Credential you created in Step 1.
Go to its Permissions tab.
Grant READ FILES permission to the Databricks user or Service Principal that will run Databricks Writer.
Setup Databricks Writer to upload stage files to GCS bucket
Step 1: Create an IAM Role on Google Cloud for Databricks Writer
In the GCP console, navigate to IAM & Admin > Roles.
Click Create Role to create a custom role.
Add the required permissions to this role. The permissions that Databricks Writer's staging operations require — are: storage.buckets.get, storage.objects.get, storage.objects.list, storage.objects.create, storage.objects.update, and storage.objects.delete. If you want Databricks Writer to automatically create the staging bucket when it doesn't already exist, also grant storage.buckets.create.
Click Create.
Step 2: Create a Service Account with the IAM Role
Navigate to IAM & Admin > Service Accounts.
Click Create Service Account. Fill in the service account name and description.
Click Create and Continue.
Note the service account’s email displayed after the creation.
Navigate to IAM & Admin > Service Accounts.
Click Grant Access.
Under New Principals, enter the service account’s email.
Under Role, select the custom role created in Step 1.
Click Save.
Step 3: Create and download a service account key
Navigate to IAM & Admin > Service Accounts and select the service account.
Go to the Keys tab.
Click Add Key > Create new key.
Choose JSON as the key type and click Create.
The key file downloads automatically. Store it securely - this file grants the permissions above. You’ll use this file later as the gcsServiceAccountKey property in Databricks Writer or in the GCS connection profile.
Step 3: grant READ FILES permission in Databricks
In your Databricks workspace, go to Catalog > Manage > Credentials and open the Storage Credential you created in Step 1.
Go to its Permissions tab.
Grant READ FILES permission to the Databricks user or Service Principal that will run Databricks Writer.
Recommendations
We recommend creating your GCS bucket in the same region as your Databricks workspace, to avoid egress charges.
If the GCS bucket doesn't already exist, Databricks Writer attempts to create it automatically — this requires the Google Cloud service account (from Step 1) to have the storage.buckets.create permission (see Step 2).
After Setup
Once you've completed the setup relevant to your cloud platform and authentication choice, you're ready to build a pipeline. See Building pipelines with Databricks Writer.