Databricks Writer
Databricks Writer writes data to Delta Lake tables in Databricks on AWS, Azure, or Google Cloud. Delta Lake is an open-source tabular storage framework that includes a transaction log to support features typically associated with relational databases, such as ACID transactions and optimistic concurrency control.
Databricks Writer can also write to Apache Iceberg tables in Databricks — see Writing to Apache Iceberg Tables.
You can use Striim's Databricks Writer to write data from transactional databases such as Oracle and SQL Server, applications such as Salesforce and ServiceNow, NoSQL databases such as Cosmos DB and MongoDB, object stores such as Amazon S3 and Google Cloud Storage, and other supported sources to Delta Lake tables in Databricks on any of the three major cloud platforms.
Summary of Databricks features common to all cloud providers
Feature | Description |
|---|---|
Supported sources | Databricks Writer accepts data from all sources supported by Striim. |
Supported write modes | Databricks Writer supports two write modes, Append Only and Merge (with an optional Optimized Merge setting within Merge mode). See Key concepts for Databricks Writer for details on when to use each. |
Additional writing features | Supports auto-quiesce after an initial load from Cosmos DB Reader, Database Reader, Mongo Cosmos DB Reader, or MongoDB Reader. Supports schema evolution to detect and propagate DDL changes from supported sources to the Delta Lake tables. Supports writing to Apache Iceberg tables in Databricks (Databricks Managed Iceberg and UniForm), in addition to Delta Lake tables. See Writing to Apache Iceberg Tables. |
Resilience and recovery | Supports connection retry to avoid application halting due to transient connection issues. Supports recovery with at-least-once processing. |
Performance | Parallel threads can increase throughput to the target in certain situations (not supported when Mode is Merge). |
Programmability | Flow Designer, TQL, and wizards in the web UI. See Building pipelines with Databricks Writer. |
Metrics and auditing | Key metrics are available through Striim's monitoring features. |
Databricks client library | Databricks JDBC driver 2.7.1. |
Key limitations | Data is written in batch mode. Streaming mode is not supported in this release. See Databricks Writer operational considerations for additional constraints. |
Configuration by cloud endpoint
Use this table to find the attributes that are specific to the cloud platform your Databricks instance runs on.
Attribute | Azure Databricks | Databricks on AWS | Databricks on Google Cloud |
|---|---|---|---|
Authentication methods |
|
|
|
Supported staging areas |
|
|
|
Staging area client libraries |
|
|
|
Unity Catalog storage credential prerequisite | Not applicable | Not applicable | Requires a Storage Credential (GCP Service Account type) created in Unity Catalog |
Note
The Service Principal (M2M OAuth) authentication method works identically regardless of which cloud your Databricks instance runs on. See Key concepts for Databricks Writer and Initial setup for Databricks Writer for the (cloud-agnostic) setup steps.
Note
Databricks File System (DBFS) is available as a staging area on all three clouds, but its use for production or sensitive data is discouraged. See What setup involves.
Note
Unity Catalog storage credential prerequisite: this requirement currently applies only to Databricks on Google Cloud (GCS staging) — see GCS Staging Setup. It does not apply to Azure or AWS staging
For more information
For Databricks on AWS:
What is Delta Lake?, Databricks on AWS, and Databricks documentation for Amazon Web Services on databricks.com
Databricks on AWS on aws.amazon.com
For Azure Databricks:
Azure Databricks on databricks.com
What is Delta Lake?, Azure Databricks, and Azure Databricks documentation on microsoft.com
For Databricks on Google Cloud:
Databricks on Google Cloud — hosted on Google Cloud's own site; Databricks does not appear to publish a dedicated GCP product page on databricks.com the way it does for AWS/Azure (see Appendix)
Databricks documentation for Google Cloud on docs.databricks.com