GCP Lakehouse Writer programmer's reference
Use this reference when you need the adapter properties, validation behavior, retry configuration, and source-to-target type mappings for GCP Lakehouse Writer.
Adapter properties
Property | Description | Required | Default or allowed value |
|---|---|---|---|
DataLakeConnectionProfileName | GCS Connection Profile for Iceberg table storage. | Yes | GCS Connection Profile |
ComputeConnectionProfileName | GCP Managed Apache Spark Connection Profile. | Yes | GCPManagedApacheSpark Connection Profile |
CatalogConnectionProfileName | GCP Lakehouse Runtime Catalog Connection Profile. | Yes | GCPLakehouseRuntimeCatalog Connection Profile |
ExternalStageConnectionProfileName | Optional GCS Connection Profile for the external staging area. | No | GCS Connection Profile |
ExternalStagingLocation | GCS folder used as the external stage. | Yes | User supplied |
IcebergTablesLocation | Warehouse location for target Iceberg tables. In REST catalog mode, use the bucket root only. Subdirectory paths are not supported. | Yes | User supplied. For REST catalog mode, use gs://<bucket> or gs://<bucket>/. |
Tables | Source-to-target table mapping. | Yes | User supplied |
ExcludedTables | Tables to exclude from wildcard mappings. | No | Blank |
Mode | Write mode. | No | APPENDONLY or MERGE |
UploadPolicy | Batching policy. | No | eventcount:100000,interval:60s |
CDDLAction | DDL handling action. | No | Process |
OptimizedMerge | Enables partial-image merge processing for CDC. | No | false |
ParallelThreads | Parallel writer threads for APPENDONLY mode. | No | 1 |
Do not configure DataLakeType, CatalogType, ComputeType, or ExternalStagingType in customer-facing examples or property tables. GCP Lakehouse Writer fixes those choices to the Google Cloud lakehouse path.
Supported data types
Before enabling initial schema creation, review the Spark-to-Iceberg data type mappings for the source. The mapping used when writing through Spark to Iceberg is not the same as the mapping used when reading Iceberg back through Spark.
For example, PostgreSQL INT2 maps to Spark short, then to Iceberg integer. When the data is read back from Iceberg through Spark, it is read as integer, not short.
The following tables list source-to-Iceberg Spark mappings by source system. Conditional mappings are shown in the Iceberg Spark target type column.
BigQuery to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
BigQuery data type | Iceberg Spark data type |
|---|---|
BIGNUMERIC | string |
BIGNUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
BIGNUMERIC(p,s) | string decimal, if (p) <= 38, if (s) <= 38 |
BOOL | boolean |
BYTES | binary |
BYTES(p) | binary |
DATE | date |
DATETIME | timestamp_ntz |
FLOAT64 | double |
GEOGRAPHY | varchar(p) |
INT64 | long |
INTERVAL | varchar(p) |
JSON | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 |
STRING | varchar(p) |
TIME | string |
TIMESTAMP | timestamp |
MySQL to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
MySQL data type | Iceberg Spark data type |
|---|---|
BIGINT UNSIGNED | Not supported |
BINARY | binary |
BINARY(p) | binary |
BIT | binary |
BIT(p) | binary |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME(s) | timestamp_ntz |
DECIMAL | decimal |
DECIMAL UNSIGNED | decimal |
DECIMAL UNSIGNED(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL UNSIGNED(p,s) | string, if (p,s) > 38, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
DOUBLE | double |
DOUBLE(p,0) | double |
DOUBLE(p,s) | double |
ENUM | string |
FLOAT | double |
FLOAT(p,s) | double, if (p,s) > 10 |
GEOMETRY | Not supported |
INT | integer |
INT UNSIGNED | integer |
JSON | string |
LONGBLOB | binary |
LONGTEXT | string |
MEDIUMBLOB | binary |
MEDIUMINT | integer |
MEDIUMINT UNSIGNED | integer |
MEDIUMTEXT | string |
SET | Not supported |
SMALLINT | short |
SMALLINT UNSIGNED | short |
TEXT | string |
TIME | string |
TIMESTAMP | timestamp_ntz |
TINYBLOB | binary |
TINYINT | byte |
TINYINT UNSIGNED | byte |
TINYTEXT | string |
VARBINARY(p) | binary |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
YEAR | integer |
Oracle to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Oracle data type | Iceberg Spark data type |
|---|---|
BFILE | Not supported |
BINARY_DOUBLE | double |
BINARY_FLOAT | float |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
CLOB | string |
FLOAT | double |
FLOAT(p) | float, if (p) <= 10 double, if (p) > 10 |
LONG | Not supported |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NCLOB | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38* |
NVARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
RAW(p) | binary |
ROWID | string |
SDO_GEOMETRY | Not supported |
UROWID | Not supported |
VARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XMLTYPE | string |
PostgreSQL to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Source data type | Iceberg Spark data type |
|---|---|
BIGSERIAL | long |
BIT | binary |
BIT(p) | binary |
BOOL | boolean |
BOX | string |
BPCHAR | char(p) |
BPCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
BYTEA | binary |
CIDR | string |
CIRCLE | string |
DATE | date |
DATERANGE | string |
FLOAT4 | float |
FLOAT8 | double |
INET | string |
INT2 | short |
INT4 | integer |
INT4RANGE | string |
INT8 | long |
INT8RANGE | string |
INTERVAL | string |
INTERVAL(p) | string |
JSON | string |
JSONB | string |
LINE | string |
LSEG | string |
MACADDR | string |
MONEY | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38* |
NUMRANGE | string |
PATH | string |
POINT | string |
POLYGON | string |
SERIAL | integer |
SMALLSERIAL | short |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | timestamp_ntz |
TIMESTAMP(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
TIMETZ | string |
TIMETZ(p) | string |
TSQUERY | string |
TSRANGE | string |
TSTZRANGE | string |
TSVECTOR | string |
TXID_SNAPSHOT | string |
UUID | string |
VARBIT | binary |
VARBIT(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Snowflake to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Snowflake data type | Iceberg Spark data type |
|---|---|
ARRAY | string |
BINARY | binary |
BINARY(p) | binary |
BOOLEAN | boolean |
DATE | date |
DOUBLE | double |
GEOGRAPHY | string |
GEOMETRY | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
OBJECT | string |
TIME | string |
TIME(p) | string |
TIMESTAMPLTZ | string |
TIMESTAMPLTZ(p) | string |
TIMESTAMPNTZ | timestamp_ntz |
TIMESTAMPNTZ(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
VARIANT | string |
Mapping notes:
Source types listed as Not supported cannot be mapped for schema creation.
Source TIME values map to string for several sources because Spark does not support the Iceberg TIME type natively.
Decimal and numeric values map to decimal only when the precision and scale are within the supported limits shown in the table. Values outside those limits may map to string.
The asterisk (*) on precision or scale conditions follows the source mapping notation.
The mapping used when data is read back from Iceberg through Spark can differ from the Spark-to-Iceberg write mapping shown here.
SQL Server to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
SQL Server data type | Iceberg Spark data type |
|---|---|
BIGINT | long |
BIGINT IDENTITY(p,s) | long, if 10 <= (p) <= 19 |
BINARY(p) | binary |
BIT | boolean |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME2 | timestamp_ntz |
DATETIME2(p) | timestamp_ntz |
DATETIMEOFFSET | timestamp |
DATETIMEOFFSET(p) | timestamp |
DECIMAL | decimal |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
FLOAT | double |
FLOAT(p) | double, if (p) > 10 |
GEOGRAPHY | Not supported |
GEOMETRY | Not supported |
HIERARCHYID | Not supported |
IMAGE | binary |
INT | integer |
INT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 long, if 10 <= (p) <= 19 |
MONEY | string |
NCHAR | char(p) |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NTEXT | string |
NUMERIC | decimal |
NUMERIC IDENTITY(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
NVARCHAR | varchar(p) |
NVARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
NVARCHAR(p) | varchar(p), if (p) <= 2147483647 |
REAL | double |
REAL(p) | double |
SMALLDATETIME | timestamp_ntz |
SMALLINT | short |
SMALLINT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 short, if 3 <= (p) <= 5 |
SMALLMONEY | string |
SQL_VARIANT | Not supported |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | binary |
TINYINT | byte |
TINYINT IDENTITY(p,s) | byte, if (p) <= 3 short, if 3 <= (p) <= 5 |
UNIQUEIDENTIFIER | varchar(p) |
VARBINARY | binary |
VARBINARY(max) | binary |
VARBINARY(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Configuration validation
Striim validates GCP Lakehouse Writer configuration at compile time before the application can deploy.
Required properties
The following properties must be non-blank:
DataLakeConnectionProfileName
ComputeConnectionProfileName
CatalogConnectionProfileName
ExternalStagingLocation
IcebergTablesLocation
Tables
CatalogConnectionProfileName is required because GCP Lakehouse Writer always uses GCP Lakehouse Runtime Catalog.
Compile-time validation messages
Condition | Compiler error |
|---|---|
One or more required properties are blank | Provided value for the following properties <list> are empty or invalid |
Named Connection Profile does not exist or cannot be retrieved | Could not retrieve the Connection Profile '<name>' provided for <property> |
Named Connection Profile exists but has the wrong type | The Connection Profile Name provided for <property> does not match the selected type for <type property>. Please provide a valid Connection Profile. |
ExternalStageConnectionProfileName is provided but the external stage type resolves blank | External Stage Connection Profile Name is provided without selecting the External Stage Type. |
IcebergTablesLocation or ExternalStagingLocation does not start with gs:// | Unsupported file scheme for the property '<property>', where the datalake type is GCS |
REST catalog mode uses a subdirectory path for IcebergTablesLocation | Use a bucket-root path such as gs://<bucket> or gs://<bucket>/. REST catalog mode does not support gs://<bucket>/<subdirectory> for IcebergTablesLocation. |
ParallelThreads is greater than 0 while Mode is MERGE | Specified configuration is not valid. Parallel Threads are allowed only in the 'APPENDONLY' mode. |
OptimizedMerge is true while Mode is APPENDONLY | Specified configuration is not valid. OptimizedMerge is supported only in the 'MERGE' mode. |
The GCP Managed Apache Spark and GCP Lakehouse Runtime Catalog Connection Profiles also perform their own profile-specific validation, such as project ID, region, and cluster name validation.
Connection retry support
Configure retry behavior on the Connection Profiles used by GCP Lakehouse Writer:
Configure compute-engine retry behavior with ConnectionRetryPolicy on the GCP Managed Apache Spark Connection Profile.
Configure GCS retry behavior with ConnectionRetryPolicy on the GCS Connection Profile used for the data lake or external staging area.
Use retry policy settings for transient connectivity issues. Retry policy does not add an Iceberg commit-conflict retry layer for concurrent writes to the same table.