GCP Lakehouse Writer operational considerations

Use these sections to monitor, troubleshoot, tune, and validate GCP Lakehouse Writer applications after deployment. For setup-time Spark sizing decisions, also review Performance guidance and best practices before you create or resize the cluster.

Monitoring

GCP Lakehouse Writer exposes table-level and adapter-level metrics in Striim. Use these metrics with Google Cloud cluster and job monitoring to understand throughput, latency, queued work, and failures.

Table write information

Table Write Info is a JSON array that lists table-level metrics.

Sub-metric	Description	Frequency
Mapped Source Table	Source table mapped to the target table.	Per batch
Last batch info	Metrics from the last executed batch.	Per batch
Last successful merge time	Time of the last executed task for a table.	Per batch
Last Applied DDL Time	Time of the last DDL executed for a table.	Per DDL batch
Last Applied DDL Statement	Last DDL statement executed for a table.	Per DDL batch
Total Batches Created	Total tasks created for a table.	Per batch
Total Batches Queued	Tasks currently queued for a table.	Per batch
Total Batches Ignored	Tasks ignored for a table.	Per batch
Total Batches Uploaded	Tasks successfully executed for a table.	Per batch
Total event info	Overall event count for a table.	Per batch
Avg Upload Time in ms	Average upload time across batches.	Per batch
Avg Compaction Time in ms	Average compaction time across batches.	Per batch
Avg In-Mem Compaction Time in ms	Average in-memory compaction time across batches.	Per batch
Avg Merge Time in ms	Average merge time across batches.	Per batch
Avg Waiting Time in Queue in ms	Average time batches spent waiting in the queue.	Per batch
Avg Event Count Per Batch	Average event count in a batch.	Per batch
Avg Batch Size in bytes	Average batch size.	Per batch
Avg Stage Resources Management Time in ms	Average time to clear or create staging resources, such as stage table and staging area.	Per batch
Avg Integration Time in ms	Average time to move a processed batch to the target table.	Per batch
Min Integration Time in ms	Minimum time for a processed batch to reach the target table.	Per batch
Max Integration Time in ms	Maximum time for a processed batch to reach the target table.	Per batch

Adapter-level metrics

Metric	Description	Frequency
Write Timestamp	Last time a batch was executed across all tables.	Per batch
Target Freshness	Time since a batch was executed across all tables.	Per batch
Discarded Event Count	Total events discarded across all tables.	Per event / batch
Connection Retry Information	Total reconnects and last known reconnect time.	When available
Queued Batches Size In Bytes	Size of all queued batches.	When available

Monitor GCP Managed Apache Spark

Use Google Cloud monitoring tools to investigate cluster and job behavior:

Open the Managed Service for Apache Spark cluster page and review cluster metrics such as memory, HDFS capacity, network traffic, CPU utilization, and disk operations.
Open Web Interfaces to access Spark History Server and YARN Resource Manager.
Open Jobs to view Managed Service for Apache Spark jobs. Click a job to inspect configuration, status, monitoring data, and logs.
Use the Spark History Server to view job status, logs, stages, executor assignment, and execution timeline.
Use YARN Resource Manager to view node status, health, available memory, and allocated applications.

When a Striim application halts because of a Spark job failure, use the job ID reported by Striim or the SHOW command to find the corresponding job in Google Cloud.

Troubleshooting

Unless otherwise noted, the following runtime conditions halt the application. If skip-on-failure is enabled, Striim deactivates the affected target table and discards the offending batch instead of halting the whole application.

DDL and schema failures

Condition	Cause	Message or symptom
Unsupported DDL received from source	Source sent RENAME TABLE, RENAME COLUMN, primary-key DDL, or constraint DDL.	Unsupported DDL operation or The DDL Operation performed is not supported
DROP COLUMN on a mapped column	Dropped column is explicitly referenced in ColumnMap.	DROP COLUMN is not supported on columns that are explicitly mentioned in the column mapper...
Key column missing or unmapped	Configured keycolumns entry does not exist or is not mapped.	Key-column-not-present or not-mapped error.
Schema not found	Namespace referenced in Tables does not exist.	Schema-not-found error.
Target table not found	Table referenced by an incoming DDL or data event cannot be found in GCP Lakehouse Runtime Catalog.	Table {<name>} not found

Storage and data lake connectivity

Condition	Message
External stage GCS bucket does not exist or is not reachable.	Unable to connect to the External Stage Location.
Iceberg tables GCS bucket does not exist or is not reachable.	Unable to connect to the Iceberg Tables Location.
Folder path inside the bucket does not exist.	File-not-found error for the configured path.
GCS service account key is missing, unreadable, or invalid.	Service-account-key error during Connection Profile validation.
GCS access is unauthorized for the configured account.	Unauthorized-access error.

Project and configuration mismatches

Condition	Message or resolution
External staging GCS project ID differs from the GCP Managed Apache Spark compute project ID.	Google project ID mismatch between external staging location and GCP Managed Apache Spark.
GCP Lakehouse Runtime Catalog project ID differs from the data lake project ID.	Verify that the GCP Lakehouse Runtime Catalog Connection Profile and GCS data lake Connection Profile point to the intended Google Cloud projects. If your deployment uses separate projects, verify the cross-project IAM configuration before restarting.

Compute job failures

Condition	Message or symptom
Spark session could not be created.	Spark session creation failed.
Spark ran out of cluster memory or cores.	The Spark execution was halted due to a runtime issue with the cluster
Catalog is not reachable from Spark.	Catalog-not-reachable error.
Catalog authentication failed after retries.	Catalog authentication error.
Invalid catalog properties or permissions.	Catalog/permission-invalid error.
Spark cannot write to the external staging area.	Staging-area-invalid error.
Compute cluster did not finish pre-run checks in time.	Pre-run-check-not-completed error.
Managed Service for Apache Spark job failed.	Root cause is populated from the job status details.
Managed Service for Apache Spark job was cancelled.	The Striim application has halted because the compute engine Spark job for this adapter is cancelled.
Managed Service for Apache Spark job did not start in time after retries.	The Striim application has halted because the compute job did not start within the expected time after exhausting all retry attempts.
GCP Managed Apache Spark service account key is invalid, unreadable, or lacks privileges.	Managed Service for Apache Spark connection-profile authentication error.

Data quality failures

Condition	Message
A NOT NULL column in the target Iceberg table receives a NULL value.	A NOT NULL column in the target Iceberg table received a NULL value
A NOT NULL target column is not mapped to any source column in MERGE mode.	A NOT NULL target column in the Iceberg table is not mapped to any source column

Common Spark errors

Error message	Likely cause	Resolution
Task was not acquired	Out-of-memory or memory pressure prevents the master node from acquiring the task. Multiple large jobs running simultaneously can cause this.	Increase worker nodes, use larger VMs, or reduce concurrent large jobs.
No agent found to be active	Out-of-memory or memory pressure makes the server or cluster unhealthy.	Stop and restart the cluster, then review concurrency and memory use.
Task not found	Cluster was deleted or stopped while a job was running.	Let jobs complete before stopping the cluster.
Driver received SIGTERM/SIGKILL signal and exited with 143 code	Spark driver on the master node ran out of memory.	Use a master node with more memory or reduce job memory pressure.

For job delays and VM out-of-memory scenarios, review the Managed Service for Apache Spark / Dataproc documentation and the Spark job logs in Google Cloud.

Performance guidance and best practices

Cluster sizing

GCP Managed Apache Spark clusters require a master node and at least one worker node. The n2-standard-4 VM type is a minimum-cost option capable of running Iceberg batches, but larger workloads may require larger VMs or more workers.

You cannot change individual VM configuration after cluster creation, so choose the master and worker machine sizes before creating the cluster. You can increase or decrease the number of worker nodes later based on workload.

Workers and parallelism

Spark performance can be increased by increasing parallelism. You can do this through:

vertical scaling: use VMs with more vCPUs, memory, and disk space
horizontal scaling: increase the number of worker nodes

Secondary workers can be added to a Dataproc cluster, but their availability is not guaranteed.

Spark configuration

Managed Service for Apache Spark determines the number of executors based on the number of workers and vCPUs per worker. You can override executor count, cores, memory, driver memory, and shuffle partitions with AdditionalConfiguration in the GCP Managed Apache Spark Connection Profile.

GCP Lakehouse Writer sets the following Spark enhancement properties to true for jobs it creates:

spark.dataproc.enhanced.optimizer.enabled
spark.dataproc.enhanced.execution.enabled

Example scenarios

Scenario 1: 1000 source tables and 8 writer instances

If a source database contains 1000 tables and 8 GCP Lakehouse Writer instances split the tables evenly, each writer handles about 125 tables. Because all 8 writers can submit batches simultaneously, the cluster should have at least 8 executor cores for task allocation.

A single worker with 8 vCPUs or two workers with 4 vCPUs each provide the same number of worker cores. However, a single n2-standard-8 worker can be preferable to two n2-standard-4 workers because less memory is consumed by Spark/system overhead and more cumulative memory is available to executors.

Scenario 2: One highly active table

For one high-traffic table, such as orders, you can optimize by using Spark scheduling and separating the high-traffic table into its own writer instance:

Create a separate Spark scheduling policy and restart the cluster.
Create one writer for the high-traffic table and another writer for other tables.
Create a GCP Managed Apache Spark Connection Profile for the high-traffic writer and add the scheduling policy in additional configuration.
Assign the new Connection Profile to the high-traffic writer.
Run both writers.

This approach can improve priority for the high-traffic table, but it can starve jobs from the other writer.

Scenario 3: Initial-load parallelism

The Spark cluster should have at least as many worker cores as the number of ParallelThreads. For example, if ParallelThreads is 8, the cluster should have either one worker with 8 vCPUs or two workers with 4 vCPUs each. More cores and memory can improve performance.

Scenario 4: Batch size

A higher batch event count can improve performance. In observed testing, a batch with 1 million operations performed up to approximately 3 times faster than a batch with 100,000 operations. Treat this as workload-dependent guidance, not a guarantee.

Before increasing eventcount, consider:

whether Spark executor memory is sufficient for the larger batch
whether other GCP Lakehouse Writer instances share the same cluster memory
whether the larger batch file increases upload time enough to affect latency

Limitations

In REST catalog mode, IcebergTablesLocation must use a root-level bucket path such as gs://my-bucket or gs://my-bucket/. Subdirectory paths such as gs://my-bucket/subdirectory are not supported.
Spark does not support the Iceberg TIME type natively. GCP Lakehouse Writer handles it as string.
GCP Lakehouse Writer does not automatically create partitioned tables. Create partitioned Iceberg tables outside Striim before writing to them.
In MERGE mode, the only supported special character in table names is underscore (_).
Some Managed Service for Apache Spark job failures require inspection of Dataproc and Spark logs in Google Cloud to identify the exact root cause.