Create a GCP Lakehouse Writer application

CREATE OR REPLACE TARGET GCPLakehouseTarget USING Global.GCPLakehouseWriter (

  DataLakeConnectionProfileName: 'admin.GCSDataLake',
  ComputeConnectionProfileName: 'admin.GCPManagedApacheSparkCP',
  CatalogConnectionProfileName: 'admin.GCPLakehouseRuntimeCatalogCP',
  IcebergTablesLocation: 'gs://my-bucket',
  ExternalStagingLocation: 'gs://my-stage-bucket/striim-stage',
  Tables: 'SRC.%,TGT.%',
  Mode: 'APPENDONLY',
  UploadPolicy: 'eventcount:100000,interval:10s',
  CDDLAction: 'Process'
)
INPUT FROM MySQLOut;

Use ParallelThreads to accelerate initial load processing when the GCP Managed Apache Spark cluster has enough executor capacity. For OLTP initial loads, configure parallel threads on both the source reader and GCP Lakehouse Writer where applicable.

The Spark cluster should have at least as many worker cores as the number of parallel writer threads.

Initial load recovery is tracked at table level. After restart, GCP Lakehouse Writer does not rewrite snapshot data into a table that already completed its initial load.

Enable OptimizedMerge only in MERGE mode. With optimized merge enabled, GCP Lakehouse Writer can process CDC events that contain only updated fields and key fields for update and delete operations. This allows source CDC readers to send partial images for update and delete operations.

MERGE mode requires a usable key for each target table. You can provide keys with the keycolumns option in the Tables property. If keycolumns is not provided, Striim uses the source table primary key when available. If neither is available, Striim concatenates all column values to identify a row.

GCP Lakehouse Writer does not implement its own retry logic for Iceberg commit conflicts. Each batch is executed as a Spark job against the target Iceberg table. The commit is governed by Iceberg optimistic concurrency behavior inside the Spark job.

If two writer instances or two applications attempt to commit to the same target table at overlapping times and Iceberg does not resolve the conflict internally, Striim surfaces a write failure.

If you are using MERGE mode, avoid configuring more than one GCP Lakehouse Writer instance or more than one Striim application to write concurrently to the same target table. If concurrent writers are unavoidable, design the flow error-handling policy with the understanding that Striim does not provide an additional commit-conflict retry layer beyond Iceberg client behavior.

Use keycolumns to control row identification in target tables.

SCOTT.EMP,mydataset.employee keycolumns(emp_num)

Wildcards can be used for the source table when all matching tables have usable key columns:

DEMO.%,mydataset.% KeyColumns(...)

If keycolumns is not provided, Striim uses the source table primary key. If neither keycolumns nor a source primary key is available, Striim concatenates all column values to create a unique row identifier.

Use ColumnMap when source and target columns do not match directly.

<source table>,<target table> ColumnMap(<target column>=<source column>,...)

Use column mapping to reorder columns or skip unmapped source columns. If a source DDL drops a column that is explicitly referenced in ColumnMap, the application halts because the mapping can no longer be satisfied.

In Striim, go to Manage Striim > Connection Profiles > Create Connection Profile.

Create a GCP Managed Apache Spark Connection Profile:

Create a GCS Connection Profile:

Create a GCP Lakehouse Runtime Catalog Connection Profile:

Deploy and run the app. The app replicates the current snapshot of the selected Oracle tables into GCS in Iceberg format. Monitor progress with Striim metrics, the SHOW command, and Google Cloud Spark job views.

The CDC application applies live inserts, updates, and deletes from Oracle to the target Iceberg tables.