Skip to main content

Key concepts for Databricks Writer

This section introduces the concepts you need to understand before configuring Databricks Writer. It covers how to build a pipeline, which write mode to use, and how Databricks Writer handles batching and schema changes. For step-by-step setup instructions, see Initial setup for Databricks Writer. For property-level detail, see Databricks Writer programmer's reference.

How Databricks Writer Works

Databricks Writer reads events from your source, batches them, and writes them to Databricks in two steps: it first stages the batched data as files in cloud storage, then issues a COPY INTO(or MERGE, for Merge mode) command that loads the staged files into the target table. This staging-then-load pattern is the same regardless of which cloud your Databricks instance runs on, which target table type you're writing to (Delta Lake, Databricks Managed Iceberg, or UniForm), or which write mode you choose.

At a high level, each Databricks Writer target does the following:

  • Reads events from its input stream (from a source, or from a CQ).

  • Batches events per target table, flushing each batch to the staging area based on the Upload Policy setting — event count or time interval, whichever comes first. See Batching and Upload Policy.

  • Loads the staged batch into the target table using COPY INTO (Append Only mode) or MERGE (Merge mode). See Choosing a Write Mode.

  • For Delta Lake tables, can optionally detect and propagate source DDL changes to the target table. See Schema Evolution. This isn't supported for Iceberg tables — see Writing to Apache Iceberg Tables.

For setup steps specific to your cloud platform and authentication method, see Initial Setup of Databricks Writer. For the full property reference, see Databricks Writer Programmer's Reference.

Three ways to build a Databricks Writer pipeline

You can create a Striim application that writes to Databricks using one of three approaches:

Wizards: Web UI wizards let you create an application from a source with Auto Schema Conversion (initial load) or CDC, selecting Databricks as the target. Wizards are the fastest way to get started when your source and use case fit the wizard's supported patterns.

Flow Designer: A visual canvas where you add and connect a source, any transformations, and a Databricks Writer target individually. Use Flow Designer when you need custom table mappings, column mappings, transformations, or multiple sources/targets in one application.

TQL: Striim's text-based configuration language. Use TQL when you want version-controlled, scriptable, or repeatable application definitions.

All three approaches configure the same underlying adapter (DeltaLakeWriter in TQL). See Building pipelines with Databricks Writer for detailed steps for each approach.

Learn about your source first

Before building a pipeline, learn the considerations specific to your source. For example, if your source is Oracle, review Striim's documentation on building pipelines from Oracle Database; if your source is ServiceNow, review the documentation on building pipelines with ServiceNow Reader. Source-specific considerations (such as CDC configuration, supplemental logging, or initial-load behavior) are outside the scope of Databricks Writer's documentation.

Choosing a write mode

Databricks Writer supports two write modes: Append Only and Merge. Optimized Merge is a performance setting available within Merge mode.

Append Only (default)

In Append Only mode, inserts, updates, and deletes from your source are all handled as inserts in the Databricks target. This means the target retains a complete history of every change, which you can query for point-in-time or historical analysis—for example, month-over-month or year-over-year reporting.

Use Append Only when: you want to preserve historical data, need an audit trail, or your reporting needs benefit from being able to query past states of the data.

Operational note:

  • For a primary key update, Append Only mode writes two records to the target — one with the previous value and one with the new value.

  • Append-only mode gives better throughput for initial load.

Merge

In Merge mode, inserts, updates, and deletes from your source are handled as inserts, updates, and deletes in the Databricks target, so the data in Databricks mirrors the data in your source.

Use Merge when: you want the Databricks tables to reflect the current source state, for operational reporting or real-time dashboards.

Required settings for Merge mode:

  • Since Delta Lake tables do not have primary keys, you should specify keycolumns in the Tables property to identify a unique-identifier column in the target table (for example: Tables:'SCOTT.EMP,mycatalog.mydatabase.employee keycolumns(emp_num)'). If you do not specify keycolumns, Striim uses the keycolumns from the source adapter's Tables property, or, absent those, concatenates all column values as a unique identifier.

  • We strongly recommend partitioning your target tables when using Merge mode, since this significantly improves performance by reducing full-table scans.

  • The Parallel Threads property setting is ignored when Mode is Merge.

Optimized Merge — an additional setting within Merge mode: set Optimized Merge to True only when Databricks Writer's input stream comes from an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events may include partial records (for example, when Oracle supplemental logging is not enabled for all columns). When your source events always include full records, leave Optimized Merge set to False (the default). Note that when Optimized Merge is True, each primary key update is handled as a separate write operation — if your source has frequent primary key updates, this can reduce write performance compared with Optimized Merge set to False.

Batching and Upload Policy

Databricks Writer does not write directly to Delta Lake tables. It first stages data as files in an external storage location (S3, ADLS Gen2, GCS, or DBFS, depending on your cloud and configuration), then issues a COPY INTO command that loads the staged files into the Databricks target table.

The Upload Policy setting controls how often Databricks Writer flushes buffered data to the staging area based on either an event count or a time interval, whichever threshold is reached first. This affects the latency between when Striim reads an event from the source and when that data becomes visible in Databricks. For exact property syntax and defaults, see Databricks Writer programmer's reference.

This staging-then-COPY INTO behavior is the same across all three clouds; only the staging area technology and its associated properties differ per cloud endpoint (see Configuration by Cloud Endpoint).

Target tables

Databricks Writer writes only to existing tables. If a source table specified in the Tables property does not exist in the target, the application halts. If you create your application using a wizard with Auto Schema Conversion, or if you enable schema evolution, Striim creates the tables for you automatically; tables created this way are not partitioned. If you plan to use Merge mode and are creating target tables yourself, we strongly recommend partitioning them, since this significantly improves performance by reducing full-table scans. You cannot partition an existing table after the fact.

Schema evolution

Databricks Writer can detect DDL changes (such as added or dropped columns) in your source and propagate them automatically to the Delta Lake target table. This behavior is controlled by the CDDL Action setting and works the same way regardless of which cloud your Databricks instance runs on. For configuration detail, see Databricks Writer programmer's reference.