Key concepts for Databricks Writer
This section introduces the concepts you need to understand before configuring Databricks Writer. It covers how to build a pipeline, which write mode to use, and how Databricks Writer handles batching and schema changes. For step-by-step setup instructions, see Initial setup for Databricks Writer. For property-level detail, see Databricks Writer programmer's reference.
How Databricks Writer Works
Databricks Writer reads events from your source, batches them, and writes them to Databricks in two steps: it first stages the batched data as files in cloud storage, then issues a COPY INTO(or MERGE, for Merge mode) command that loads the staged files into the target table. This staging-then-load pattern is the same regardless of which cloud your Databricks instance runs on, which target table type you're writing to (Delta Lake, Databricks Managed Iceberg, or UniForm), or which write mode you choose.
At a high level, each Databricks Writer target does the following:
Reads events from its input stream (from a source, or from a CQ).
Batches events per target table, flushing each batch to the staging area based on the Upload Policy setting — event count or time interval, whichever comes first. See Batching and Upload Policy.
Loads the staged batch into the target table using COPY INTO (Append Only mode) or MERGE (Merge mode). See Choosing a Write Mode.
For Delta Lake tables, can optionally detect and propagate source DDL changes to the target table. See Schema Evolution. This isn't supported for Iceberg tables — see Writing to Apache Iceberg Tables.
For setup steps specific to your cloud platform and authentication method, see Initial Setup of Databricks Writer. For the full property reference, see Databricks Writer Programmer's Reference.
How Databricks Writer Works
Databricks Writer reads events from your source, batches them, and writes them to Databricksin two steps: it first stages the batched data as files in staging area, then issues SQL commands that load the staged files into the target table. This staging-then-load pattern is the same regardless of which cloud your Databricks instance runs on, which target table type you're writing to (Delta Lake, Databricks-managed Iceberg, or UniForm), or which write mode you choose.
At a high level, each Databricks Writer target does the following:
Reads events from its input stream (from a source, or from a CQ).
Batches events per target table, flushing each batch to the staging area based on the Upload Policy setting — event count or time interval, whichever comes first. See Batching and Upload Policy.
Loads the staged batch into the target table using COPY INTO (Append Only mode) or MERGE (Merge mode). See Choosing a write mode.
For Delta Lake tables, can optionally detect and propagate source DDL changes to the target table. See Schema Evolution. This isn't supported for Iceberg tables — see Writing to Apache Iceberg tables.
For setup steps specific to your cloud platform and authentication method, see Initial setup for Databricks Writer. For the full property reference, see Databricks Writer programmer's reference.
Three ways to build a Databricks Writer pipeline
You can create a Striim application that writes to Databricks using one of three approaches:
Wizards: Web UI wizards let you create an application from a source with Auto Schema Conversion (initial load) or CDC, selecting Databricks as the target. Wizards are the fastest way to get started when your source and use case fit the wizard's supported patterns.
Flow Designer: A visual canvas where you add and connect a source, any transformations, and a Databricks Writer target individually. Use Flow Designer when you need custom table mappings, column mappings, transformations, or multiple sources/targets in one application.
TQL: Striim's text-based configuration language. Use TQL when you want version-controlled, scriptable, or repeatable application definitions.
All three approaches configure the same underlying adapter (DeltaLakeWriter in TQL). See Building pipelines with Databricks Writer for detailed steps for each approach.
Learn about your source first
Before building a pipeline, learn the considerations specific to your source. For example, if your source is Oracle, review Striim's documentation on building pipelines from Oracle Database; if your source is ServiceNow, review the documentation on building pipelines with ServiceNow Reader. Source-specific considerations (such as CDC configuration, supplemental logging, or initial-load behavior) are outside the scope of Databricks Writer's documentation.
Choosing a write mode
Databricks Writer supports two write modes: Append Only and Merge. Optimized Merge is a performance setting available within Merge mode.
Append Only (default)
In Append Only mode, inserts, updates, and deletes from your source are all handled as inserts in the Databricks target. This means the target retains a complete history of every change, which you can query for point-in-time or historical analysis—for example, month-over-month or year-over-year reporting.
Use Append Only when: you want to preserve historical data, need an audit trail, or your reporting needs benefit from being able to query past states of the data.
Operational note:
For a primary key update, Append Only mode writes two records to the target — one with the previous value and one with the new value.
Append-only mode gives better throughput for initial load.
Merge
In Merge mode, inserts, updates, and deletes from your source are handled as inserts, updates, and deletes in the Databricks target, so the data in Databricks mirrors the data in your source.
Use Merge when: you want the Databricks tables to reflect the current source state, for operational reporting or real-time dashboards.
Required settings for Merge mode:
Since Delta Lake tables do not have primary keys, you should specify
keycolumnsin theTablesproperty to identify a unique-identifier column in the target table (for example:Tables:'SCOTT.EMP,mycatalog.mydatabase.employee keycolumns(emp_num)'). If you do not specifykeycolumns, Striim uses the keycolumns from the source adapter'sTablesproperty, or, absent those, concatenates all column values as a unique identifier.We strongly recommend partitioning your target tables when using Merge mode, since this significantly improves performance by reducing full-table scans.
The Parallel Threads property setting is ignored when Mode is Merge.
Optimized Merge — an additional setting within Merge mode: set Optimized Merge to True only when Databricks Writer's input stream comes from an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events may include partial records (for example, when Oracle supplemental logging is not enabled for all columns). When your source events always include full records, leave Optimized Merge set to False (the default). Note that when Optimized Merge is True, each primary key update is handled as a separate write operation — if your source has frequent primary key updates, this can reduce write performance compared with Optimized Merge set to False.
Batching and Upload Policy
Databricks Writer does not write directly to Delta Lake tables. It first stages data as files in an external storage location (S3, ADLS Gen2, GCS, or DBFS, depending on your cloud and configuration), then issues a COPY INTO command that loads the staged files into the Databricks target table.
The Upload Policy setting controls how often Databricks Writer flushes buffered data to the staging area based on either an event count or a time interval, whichever threshold is reached first. This affects the latency between when Striim reads an event from the source and when that data becomes visible in Databricks. For exact property syntax and defaults, see Databricks Writer programmer's reference.
This staging-then-COPY INTO behavior is the same across all three clouds; only the staging area technology and its associated properties differ per cloud endpoint (see Configuration by Cloud Endpoint).
Sizing Event Count and Interval
Databricks Writer batches incoming events per target table — each table Databricks Writer writes to has its own batch. A batch is flushed to the staging area when either the Upload Policy event count or interval threshold is reached, whichever happens first.
Event count sets a ceiling on how large a batch can grow. A high event count suits high-volume workloads, since it produces fewer, larger stage files — but very large stage files increase memory use on the Striim server. A low event count produces more, smaller stage files, which suits low-volume workloads, but can reduce Merge mode performance against large target tables.
Interval sets a time-based backstop. A high interval lets batches accumulate longer before flushing, which is useful when you want less-frequent merges — but produces large stage files if event count is also high. A low interval flushes more frequently regardless of traffic, keeping data timely during low-volume periods, but produces more stage files for the same number of events.
The default Upload Policy (eventcount:100000, interval:60s) works well for most initial-load and CDC workloads. For CDC against large tables under heavy traffic, consider a higher event count — this produces larger stage files and uses more Striim server memory. For bursty traffic, combine a high enough event count to absorb a burst into a single batch with a low enough interval to flush data during lulls, so it doesn't sit buffered indefinitely.
When Optimized Merge is enabled (Mode is Merge), a primary key update on a table forces that table's current batch to flush immediately, ahead of the configured event count or interval.
Target tables
Databricks Writer writes only to existing tables. If a source table specified in the Tables property does not exist in the target, the application halts. If you create your application using a wizard with Auto Schema Conversion, or if you enable schema evolution, Striim creates the tables for you automatically; tables created this way are not partitioned. If you plan to use Merge mode and are creating target tables yourself, we strongly recommend partitioning them, since this significantly improves performance by reducing full-table scans. You cannot partition an existing table after the fact.
Schema evolution
Databricks Writer can detect DDL changes (such as added or dropped columns) in your source and propagate them automatically to the Delta Lake target table. This behavior is controlled by the CDDL Action setting and works the same way regardless of which cloud your Databricks instance runs on. For configuration detail, see Databricks Writer programmer's reference.
Writing to Apache Iceberg tables
Databricks Writer can write to existing Apache Iceberg tables in Databricks. No special Databricks Writer configuration is required. The tables must be created and configured as described in Read Delta tables with Iceberg clients. Iceberg tables cannot be created using a Striim initial load wizard and schema evolution (CDDL) is not supported.
Databricks Writer can write to two kinds of Apache Iceberg tables in Databricks, in addition to Delta Lake tables:
Databricks Managed Iceberg tables (Format Version 2) — native Iceberg tables created with USING iceberg in Unity Catalog.
UniForm tables — Delta Lake tables with Iceberg v2 metadata generated automatically (delta.enableIcebergCompatV2 = true); Databricks Writer writes to the underlying Delta table, and Iceberg-compatible clients can read the same data.
Databricks Writer supports only insert, update, and delete DML operations on Iceberg tables. Initial schema creation is not supported: the target tables must already exist before you deploy your application. Schema evolution is not supported. See Setup for Apache Iceberg Tables.
Where data lands depends entirely on how the target table was created, not on anything Databricks Writer configures: if the table's schema or catalog was created against an external location, data is stored there; if it uses Databricks-managed storage, data is stored in the default cloud storage Databricks provisions for the workspace. UniForm tables follow the same rule.Databricks Managed Iceberg and UniForm tables use the same write modes, staging areas (including GCS), authentication methods, and Databricks permissions as Delta Lake tables — see Choosing a Write Mode above, Initial Setup of Databricks Writer, and Databricks Permissions Required for Any Authentication Method.