Skip to main content

Building pipelines with Databricks Writer

This section shows you how to build a Striim application that writes to Databricks, using a wizard, Flow Designer, or TQL. For the concepts behind these approaches and how to choose a write mode, see Key concepts for Databricks Writer. For complete, worked examples per cloud endpoint, see Databricks Writer examples.

Prerequisites for Databricks Writer applications

Before creating a Databricks target in a Striim application:

  • Complete the setup described in Initial setup for Databricks Writer for your cloud platform and chosen authentication method.

  • Complete any setup tasks required for your application's source (for example, Oracle CDC prerequisites).

  • Know the property values required for your authentication method — a Personal Access Token, or Service Principal Client ID/Client Secret/Connection URL, or (Azure only) Entra ID Client ID/Client Secret/Refresh Token/Tenant ID.

  • Choose a write mode — see Key concepts for Databricks Writer. If you choose Merge mode, note that the Parallel Threads property is ignored.

  • If you're building your application in TQL and don't plan to use schema evolution, create the target tables in Databricks first. This isn't necessary if you use schema evolution, or if you build your application using a wizard with Auto Schema Conversion, since Striim creates the tables automatically in those cases.

Building an application using a wizard

Web UI wizards let you create an application from a source with Auto Schema Conversion (initial load) or CDC, with Databricks Writer as the target. When you create a Databricks target using a wizard, you specify your Connection URL and authentication credentials (Personal Access Token, or Service Principal Client ID/Client Secret), and the wizard sets the Tables property based on the source tables you select.

Note

Table naming with Unity Catalog: this release is built around Databricks workspaces with Unity Catalog enabled, where target tables use three-part names (catalog.database.table). Confirm with your Databricks administrator whether your workspace uses Unity Catalog or the legacy Hive metastore, since this affects how source tables map to target table names during wizard-driven table creation

Using the automated pipeline wizard

Databricks Writer is supported as a target in Striim's automated pipeline wizards. An automated pipeline wizard creates two linked applications — one to perform an initial load of existing source data, and a second to continuously apply changes to the target via CDC — with handover between the two handled automatically. Recovery is enabled automatically for both applications (except for initial load from a Salesforce source, which is a general automated-pipeline limitation, not specific to Databricks).

Schema creation: the automated pipeline wizard automatically creates the corresponding schema in Databricks — you don't need to create target tables yourself first when using this wizard.

Catalog scope: an automated pipeline wizard writes to only one catalog. The wizard takes a Catalog Name as an input:

  • If the destination is a Unity Catalog, provide the catalog name.

  • If the destination is a legacy Hive metastore, you can leave this field empty.

If you need to write to multiple catalogs, create a separate automated pipeline wizard for each catalog — one wizard cannot span multiple catalogs.

For general information about automated pipelines, see Introducing automated pipelines

Building an application using Flow Designer

Flow Designer gives you a visual canvas for adding a source, any transformations, and a Databricks Writer target, and connecting them individually. Use Flow Designer instead of a wizard when you need custom table or column mappings, need to combine multiple sources, or need to filter or transform data before it reaches Databricks.

Configure the Databricks Writer target's properties — connection/authentication, target catalog/database/table (via the Tables property), write mode, staging area, and schema evolution setting — as described in Databricks Writer programmer's reference.

Building an application using TQL

TQL lets you define your application as version-controlled, scriptable text. Databricks Writer is invoked in TQL as the Global.DeltaLakeWriter adapter, as a TARGET that reads from an INPUT FROM stream (typically the output of a source or a CQ).

Basic syntax:

CREATE TARGET <target_name> USING Global.DeltaLakeWriter (
  <property>: '<value>',
  ...
)
INPUT FROM <stream_name>;

Minimal example — Append Only mode with a Personal Access Token:

CREATE TARGET DatabricksAppendOnly USING DeltaLakeWriter (
  personalAccessToken: '<personal-access-token>',
  tables: 'mydb.employee,mydatabase.employee',
  stageLocation: '/StriimStage/',
  connectionUrl: 'jdbc:databricks://<workspace-hostname>:443/default;transportMode=http;ssl=1;httpPath=<http-path>;AuthMech=3;UID=token;'
)
INPUT FROM ns1.sourceStream;

Minimal example — Merge mode with Optimized Merge:

CREATE TARGET DatabricksAppendOnly USING DeltaLakeWriter (
  personalAccessToken: '<personal-access-token>',
  tables: 'mydb.employee,mydatabase.employee',
  stageLocation: '/StriimStage/',
  connectionUrl: 'jdbc:databricks://<workspace-hostname>:443/default;transportMode=http;ssl=1;httpPath=<http-path>;AuthMech=3;UID=token;',
  mode: 'MERGE',
  optimizedMerge: 'true'
)
INPUT FROM ns1.sourceStream;

For complete application-level examples (including a source, GCS staging, and Service Principal authentication) for each cloud endpoint, see Databricks Writer examples.

After you deploy

Once your application is deployed and running, Databricks Writer stages incoming data and periodically loads data into your target tables. For monitoring, troubleshooting, and operational behavior, including what happens on application restart or failure, see Databricks Writer operational considerations.