GCP Lakehouse Writer programmer's reference
Use this reference when you need the adapter properties, validation behavior, retry configuration, and source-to-target type mappings for GCP Lakehouse Writer.
GCP Lakehouse Writer properties
property | type | default value | notes |
|---|---|---|---|
Catalog Connection Profile Name | String |
| |
CDDL Action | String | Process | |
Compute Connection Profile Name | String |
| |
Data Lake Connection Profile Name | String |
| |
Excluded Tables | String |
| If |
External Stage Connection Profile Name | String |
| |
External Staging Location | String |
| GCS folder used as the external stage. |
Iceberg Tables Location | String | Warehouse location for target Iceberg tables. In REST catalog mode, use the bucket root only. Subdirectory paths are not supported. User supplied. For REST catalog mode, use | |
Mode | enum | APPENDONLY | With the default setting of APPENDONLY, update and delete operations in the source are handled as inserts in the target. Use this mode for initial loads, audit tables, or non-CDC sources. Set to MERGE to handle update and delete operations in the source as updates and deletes in the target., so the target replicates the source. |
Optimized Merge | Boolean | False | Appears in Flow Designer only when Mode is Merge. Set to True only when Mode is MERGE and the target's input stream is the output of an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events will include partial records. For example, with Oracle Reader, when supplemental logging has not been enabled for all columns, partial records are sent for updates. When the source events will always include full records, leave this set to False. When Optimized Merge is True, each primary key update is handled in a separate write operation. If the source has frequent primary key updates, this may lead to a decline in write performance compared with Optimized Merge = False. |
Parallel Threads | Integer | 1 | Not supported when Mode is Merge. |
Tables | String | User supplied | See Table mapping. |
Upload Policy | String | eventcount:100000,interval:60s | The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Cached data is written to the storage account every time any of the specified values is exceeded. With the default value, data will be written every five minutes or sooner if the cache contains 10,000 events. When the app is undeployed, all remaining data is written to the storage account. |
Supported data types
Before enabling initial schema creation, review the Spark-to-Iceberg data type mappings for the source. The mapping used when writing through Spark to Iceberg is not the same as the mapping used when reading Iceberg back through Spark.
For example, PostgreSQL INT2 maps to Spark short, then to Iceberg integer. When the data is read back from Iceberg through Spark, it is read as integer, not short.
The following tables list source-to-Iceberg Spark mappings by source system. Conditional mappings are shown in the Iceberg Spark target type column.
BigQuery to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
BigQuery data type | Iceberg Spark data type |
|---|---|
BIGNUMERIC | string |
BIGNUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
BIGNUMERIC(p,s) | string decimal, if (p) <= 38, if (s) <= 38 |
BOOL | boolean |
BYTES | binary |
BYTES(p) | binary |
DATE | date |
DATETIME | timestamp_ntz |
FLOAT64 | double |
GEOGRAPHY | varchar(p) |
INT64 | long |
INTERVAL | varchar(p) |
JSON | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 |
STRING | varchar(p) |
TIME | string |
TIMESTAMP | timestamp |
MySQL to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
MySQL data type | Iceberg Spark data type |
|---|---|
BIGINT UNSIGNED | Not supported |
BINARY | binary |
BINARY(p) | binary |
BIT | binary |
BIT(p) | binary |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME(s) | timestamp_ntz |
DECIMAL | decimal |
DECIMAL UNSIGNED | decimal |
DECIMAL UNSIGNED(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL UNSIGNED(p,s) | string, if (p,s) > 38, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
DOUBLE | double |
DOUBLE(p,0) | double |
DOUBLE(p,s) | double |
ENUM | string |
FLOAT | double |
FLOAT(p,s) | double, if (p,s) > 10 |
GEOMETRY | Not supported |
INT | integer |
INT UNSIGNED | integer |
JSON | string |
LONGBLOB | binary |
LONGTEXT | string |
MEDIUMBLOB | binary |
MEDIUMINT | integer |
MEDIUMINT UNSIGNED | integer |
MEDIUMTEXT | string |
SET | Not supported |
SMALLINT | short |
SMALLINT UNSIGNED | short |
TEXT | string |
TIME | string |
TIMESTAMP | timestamp_ntz |
TINYBLOB | binary |
TINYINT | byte |
TINYINT UNSIGNED | byte |
TINYTEXT | string |
VARBINARY(p) | binary |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
YEAR | integer |
Oracle to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Oracle data type | Iceberg Spark data type |
|---|---|
BFILE | Not supported |
BINARY_DOUBLE | double |
BINARY_FLOAT | float |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
CLOB | string |
FLOAT | double |
FLOAT(p) | float, if (p) <= 10 double, if (p) > 10 |
LONG | Not supported |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NCLOB | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38* |
NVARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
RAW(p) | binary |
ROWID | string |
SDO_GEOMETRY | Not supported |
UROWID | Not supported |
VARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XMLTYPE | string |
PostgreSQL to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Source data type | Iceberg Spark data type |
|---|---|
BIGSERIAL | long |
BIT | binary |
BIT(p) | binary |
BOOL | boolean |
BOX | string |
BPCHAR | char(p) |
BPCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
BYTEA | binary |
CIDR | string |
CIRCLE | string |
DATE | date |
DATERANGE | string |
FLOAT4 | float |
FLOAT8 | double |
INET | string |
INT2 | short |
INT4 | integer |
INT4RANGE | string |
INT8 | long |
INT8RANGE | string |
INTERVAL | string |
INTERVAL(p) | string |
JSON | string |
JSONB | string |
LINE | string |
LSEG | string |
MACADDR | string |
MONEY | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38* |
NUMRANGE | string |
PATH | string |
POINT | string |
POLYGON | string |
SERIAL | integer |
SMALLSERIAL | short |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | timestamp_ntz |
TIMESTAMP(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
TIMETZ | string |
TIMETZ(p) | string |
TSQUERY | string |
TSRANGE | string |
TSTZRANGE | string |
TSVECTOR | string |
TXID_SNAPSHOT | string |
UUID | string |
VARBIT | binary |
VARBIT(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Snowflake to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
Snowflake data type | Iceberg Spark data type |
|---|---|
ARRAY | string |
BINARY | binary |
BINARY(p) | binary |
BOOLEAN | boolean |
DATE | date |
DOUBLE | double |
GEOGRAPHY | string |
GEOMETRY | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
OBJECT | string |
TIME | string |
TIME(p) | string |
TIMESTAMPLTZ | string |
TIMESTAMPLTZ(p) | string |
TIMESTAMPNTZ | timestamp_ntz |
TIMESTAMPNTZ(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
VARIANT | string |
Mapping notes:
Source types listed as Not supported cannot be mapped for schema creation.
Source TIME values map to string for several sources because Spark does not support the Iceberg TIME type natively.
Decimal and numeric values map to decimal only when the precision and scale are within the supported limits shown in the table. Values outside those limits may map to string.
The asterisk (*) on precision or scale conditions follows the source mapping notation.
The mapping used when data is read back from Iceberg through Spark can differ from the Spark-to-Iceberg write mapping shown here.
SQL Server to Iceberg Spark
Note
p stands for Precision and s stands for Scale.
SQL Server data type | Iceberg Spark data type |
|---|---|
BIGINT | long |
BIGINT IDENTITY(p,s) | long, if 10 <= (p) <= 19 |
BINARY(p) | binary |
BIT | boolean |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME2 | timestamp_ntz |
DATETIME2(p) | timestamp_ntz |
DATETIMEOFFSET | timestamp |
DATETIMEOFFSET(p) | timestamp |
DECIMAL | decimal |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
FLOAT | double |
FLOAT(p) | double, if (p) > 10 |
GEOGRAPHY | Not supported |
GEOMETRY | Not supported |
HIERARCHYID | Not supported |
IMAGE | binary |
INT | integer |
INT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 long, if 10 <= (p) <= 19 |
MONEY | string |
NCHAR | char(p) |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NTEXT | string |
NUMERIC | decimal |
NUMERIC IDENTITY(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | string, if (p,s) > 38* string, if (s) > 38* decimal, if (p) <= 38, if (s) <= 38 |
NVARCHAR | varchar(p) |
NVARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
NVARCHAR(p) | varchar(p), if (p) <= 2147483647 |
REAL | double |
REAL(p) | double |
SMALLDATETIME | timestamp_ntz |
SMALLINT | short |
SMALLINT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 short, if 3 <= (p) <= 5 |
SMALLMONEY | string |
SQL_VARIANT | Not supported |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | binary |
TINYINT | byte |
TINYINT IDENTITY(p,s) | byte, if (p) <= 3 short, if 3 <= (p) <= 5 |
UNIQUEIDENTIFIER | varchar(p) |
VARBINARY | binary |
VARBINARY(max) | binary |
VARBINARY(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Configuration validation
Striim validates GCP Lakehouse Writer configuration at compile time before the application can deploy.
Required properties
The following properties must be non-blank:
DataLakeConnectionProfileName
ComputeConnectionProfileName
CatalogConnectionProfileName
ExternalStagingLocation
IcebergTablesLocation
Tables
CatalogConnectionProfileName is required because GCP Lakehouse Writer always uses GCP Lakehouse Runtime Catalog.
Compile-time validation messages
Condition | Compiler error |
|---|---|
One or more required properties are blank | Provided value for the following properties <list> are empty or invalid |
Named Connection Profile does not exist or cannot be retrieved | Could not retrieve the Connection Profile '<name>' provided for <property> |
Named Connection Profile exists but has the wrong type | The Connection Profile Name provided for <property> does not match the selected type for <type property>. Please provide a valid Connection Profile. |
ExternalStageConnectionProfileName is provided but the external stage type resolves blank | External Stage Connection Profile Name is provided without selecting the External Stage Type. |
IcebergTablesLocation or ExternalStagingLocation does not start with gs:// | Unsupported file scheme for the property '<property>', where the datalake type is GCS |
REST catalog mode uses a subdirectory path for IcebergTablesLocation | Use a bucket-root path such as gs://<bucket> or gs://<bucket>/. REST catalog mode does not support gs://<bucket>/<subdirectory> for IcebergTablesLocation. |
ParallelThreads is greater than 0 while Mode is MERGE | Specified configuration is not valid. Parallel Threads are allowed only in the 'APPENDONLY' mode. |
OptimizedMerge is true while Mode is APPENDONLY | Specified configuration is not valid. OptimizedMerge is supported only in the 'MERGE' mode. |
The GCP Managed Apache Spark and GCP Lakehouse Runtime Catalog Connection Profiles also perform their own profile-specific validation, such as project ID, region, and cluster name validation.
Connection retry support
Configure retry behavior on the Connection Profiles used by GCP Lakehouse Writer:
Configure compute-engine retry behavior with ConnectionRetryPolicy on the GCP Managed Apache Spark Connection Profile.
Configure GCS retry behavior with ConnectionRetryPolicy on the GCS Connection Profile used for the data lake or external staging area.
Use retry policy settings for transient connectivity issues. Retry policy does not add an Iceberg commit-conflict retry layer for concurrent writes to the same table.