S3 Tables Writer programmer's reference
This reference describes every property exposed by S3TablesIcebergWriter and by its four connection profiles, plus the data type mapping, DDL support, and special-character rules that govern how source data is represented in S3 Tables.
S3 Tables Writer adapter properties
Property | Type / default | Description |
|---|---|---|
CatalogConnectionProfileName | String, optional | Connection profile with the catalog details; the catalog endpoint must be accessible from the compute engine. Leave empty when CatalogType is S3Tables. Supported connection profile type: AWSGlueCatalog. |
CatalogType | S3TablesCatalogType (enum), default S3Tables | The catalog used by the Iceberg warehouse. Values: S3Tables (default) or AWSGlueCatalog. |
CDDLAction | CDDLActions (enum), default Process | How to act on DDL events: Process (apply at target), Halt (halt the pipeline on any DDL event), or Ignore (drop the DDL and keep the pipeline running). |
ComputeConnectionProfileName | String, required | Connection profile with the compute engine details used to write data in Iceberg format. Supported connection profile type: AWSEMR. |
DataLakeConnectionProfileName | String, required | Connection profile with the S3 Tables data lake details. Supported connection profile type: S3Tables. |
ExcludedTables | String, optional | Tables excluded from the set of interested tables — data is not written to these. |
ExternalStageConnectionProfileName | String, required | Connection profile with the S3 cloud storage details used as an external stage area. Supported connection profile type: S3. |
ExternalStagingLocation | String, required | Path to a folder in the S3 bucket used as the staging area — hosts batch data, metadata, and metrics files during execution; cleared when the adapter closes. The path must start with the bucket name (see Staging path scheme). |
Mode | DWHWriteModes (enum), default APPENDONLY | Write mode: MERGE or APPENDONLY. |
OptimizedMerge | Boolean, default false | When true, S3 Tables Writer can process partial updates (key fields plus only the changed values) for update and delete operations. When false, all field values must be present. |
ParallelThreads | Integer, optional | Number of parallel threads used during APPENDONLY mode to parallelize snapshot loads. |
Tables | String, required | Target tables and their mapped source tables, fully qualified. Key column information can also be provided (see Key column support). |
UploadPolicy | String, default eventcount:100000,interval:60s | Determines when cached data is written to S3 storage, based on event count and time interval (see Upload policy parameters). |
Connection profiles
AWS EMR connection profile
Endpoint name: AWS EMR
Property | Type / default | Description | Alterable |
|---|---|---|---|
AccessKey | String, optional | Optional AWS access key for authenticating with AWS services. If not provided, uses the default AWS credentials chain or IAM role-based authentication. | Yes |
AdditionalConfiguration | String, optional | Additional Spark/Hadoop configuration properties in key=value format, applied to EMR jobs. | No |
ClientConfiguration | String, default MaxErrorRetry=5 | SDK-level retry settings. | No |
ClusterID | String, required | The unique identifier of the AWS EMR cluster used as the compute engine for Iceberg operations. | No |
JarLocations | String, optional | Custom JAR file locations (S3 or local paths) to include in Spark job execution. Multiple locations can be provided. | No |
Region | String, required | The AWS region where the EMR cluster is deployed (for example, us-east-1). | No |
SecretKey | Password, optional | Optional AWS secret key corresponding to the access key, stored securely. If not provided, uses the default AWS credentials chain or IAM role-based authentication. | Yes |
S3 Tables connection profile
Endpoint name: S3 Tables
Property | Type / default | Description | Alterable |
|---|---|---|---|
AdditionalConfiguration | String, optional | Additional S3 Tables-specific configuration properties in key=value format, applied when connecting to the bucket. | No |
Region | String, required | The AWS region where the S3 Tables bucket is deployed. | No |
S3TablesBucketARN | String, required | The full ARN of the S3 Tables bucket used as the data lake. Format: arn:aws:s3tables:region:account-id:bucket/bucket-name. | No |
Note: S3TablesBucketARN can be found in the S3 Tables bucket's properties in the AWS Console. Authentication to S3 Tables happens exclusively through the IAM role attached to the EMR cluster's instance profile. Access Key and Secret Key are not exposed as configurable properties on this connection profile. This is intentional design, not a defect.
AWS Glue Catalog connection profile
Endpoint name: AWS Glue Catalog
Property | Type / default | Description | Alterable |
|---|---|---|---|
AccountID | String, required | The 12-digit AWS account ID that owns the Glue Catalog. | No |
AdditionalConfiguration | String, optional | Additional Glue Catalog configuration properties in key=value format (for example, catalog-encryption-mode=DISABLED). | No |
region | String, required, lowercase | The AWS region where the Glue Catalog is deployed. | No |
Note: Authentication to Glue happens exclusively through the same EMR instance-profile IAM role used for S3 Tables, which must additionally be granted the Glue-specific permissions and the Lake Formation grant described in S3 Tables Writer initial setup. Access Key and Secret Key are not exposed as configurable properties on this connection profile.
S3 connection profile (for external staging)
Endpoint name: S3
Property | Type / default | Description | Alterable |
|---|---|---|---|
s3AccessKey | Password, optional | Optional AWS access key for S3 authentication, stored securely. If not provided, uses the default AWS credentials chain or IAM role-based authentication. | Yes |
s3BucketName | String, default striim-bucket — not functional | Not used to resolve the external stage bucket — the bucket is always derived from the writer's ExternalStagingLocation property instead. Planned for removal. | No |
s3Region | String, default us-west-1 | The AWS region where the S3 staging bucket is deployed. | No |
s3SecretAccessKey | Password, optional | Optional AWS secret key corresponding to the access key, stored securely. | Yes |
Note: If s3AccessKey and s3SecretAccessKey are not provided, the adapter uses IAM credentials configured in the environment. Altering s3AccessKey or s3SecretAccessKey on a running application takes effect on the next connection retry.
Initial schema creation and data type mapping
S3 Tables Writer creates the corresponding namespaces and tables automatically during initial load if they do not already exist.
Enable CreateSchema on the source adapter (for example DatabaseReader) when configuring an initial load pipeline. Auto schema creation is available from any RDBMS source that connects through DatabaseReader over JDBC, including Oracle, SQL Server, MySQL, and PostgreSQL.
During the snapshot load, DDL statements for creating namespaces, followed by tables, are sent to S3 Tables Writer, which executes them one after another on the compute engine. Once all DDL statements are executed, the snapshot data follows.
Schema Conversion Utility
The Schema Conversion command-line utility generates the initial target schema from source connection details, without running a full Striim application:
bin/schemaConversionUtility.sh -s=oracle -d="jdbc:oracle:thin:@//<host>:<port>/<sid>" -u="<username>" -p="<password>" -b='<schema>.<TABLE>;' -t="spark"
Parameter | Description |
|---|---|
-s | Source database type. |
-d | JDBC URL to the source database. |
-u | Username. |
-p | Password. |
-b | Semicolon-separated list of source tables to generate a schema for. |
-t | Target type: spark. |
The utility returns SQL files containing target-compatible DDL for creating S3 Tables, which you can execute directly with any Spark engine or the AWS Athena query console.
Data type mapping
S3 Tables Writer maps each source column's data type to a Spark data type when writing to Iceberg. The following tables list, for each supported source, which source data types map to each Spark data type. Where a mapping depends on the column's declared precision or scale, the condition is shown in parentheses.
BigQuery
Source data type | Spark data type |
|---|---|
BIGNUMERIC | string |
BIGNUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
BIGNUMERIC(p,s) | string decimal, if (p) <= 38, if (s) <= 38 |
BOOL | boolean |
BYTES | binary |
BYTES(p) | binary |
DATE | date |
DATETIME | timestamp_ntz |
FLOAT64 | double |
GEOGRAPHY | varchar(p) |
INT64 | long |
INTERVAL | varchar(p) |
JSON | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 |
STRING | varchar(p) |
TIME | string |
TIMESTAMP | timestamp |
MySQL
Source data type | Spark data type |
|---|---|
BIGINT UNSIGNED | Not supported |
BINARY | binary |
BINARY(p) | binary |
BIT | binary |
BIT(p) | binary |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME(s) | timestamp_ntz |
DECIMAL | decimal |
DECIMAL UNSIGNED | decimal |
DECIMAL UNSIGNED(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL UNSIGNED(p,s) | string, if (p,s) > 38, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
DOUBLE | double |
DOUBLE(p,0) | double |
DOUBLE(p,s) | double |
ENUM | string |
FLOAT | double |
FLOAT(p,s) | double, if (p,s) > 10 |
GEOMETRY | Not supported |
INT | integer |
INT UNSIGNED | integer |
JSON | string |
LONGBLOB | binary |
LONGTEXT | string |
MEDIUMBLOB | binary |
MEDIUMINT | integer |
MEDIUMINT UNSIGNED | integer |
MEDIUMTEXT | string |
SET | Not supported |
SMALLINT | short |
SMALLINT UNSIGNED | short |
TEXT | string |
TIME | string |
TIMESTAMP | timestamp_ntz |
TINYBLOB | binary |
TINYINT | byte |
TINYINT UNSIGNED | byte |
TINYTEXT | string |
VARBINARY(p) | binary |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
YEAR | integer |
Oracle
Source data type | Spark data type |
|---|---|
BFILE | Not supported |
BINARY_DOUBLE | double |
BINARY_FLOAT | float |
BLOB | binary |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
CLOB | string |
FLOAT | double |
FLOAT(p) | float, if (p) <= 10 double, if (p) > 10 |
LONG | Not supported |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NCLOB | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38 |
NVARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
RAW(p) | binary |
ROWID | string |
SDO_GEOMETRY | Not supported |
UROWID | Not supported |
VARCHAR2(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XMLTYPE | string |
PostgreSQL
Source data type | Spark data type |
|---|---|
BIGSERIAL | long |
BIT | binary |
BIT(p) | binary |
BOOL | boolean |
BOX | string |
BPCHAR | char(p) |
BPCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
BYTEA | binary |
CIDR | string |
CIRCLE | string |
DATE | date |
DATERANGE | string |
FLOAT4 | float |
FLOAT8 | double |
INET | string |
INT2 | short |
INT4 | integer |
INT4RANGE | string |
INT8 | long |
INT8RANGE | string |
INTERVAL | string |
INTERVAL(p) | string |
JSON | string |
JSONB | string |
LINE | string |
LSEG | string |
MACADDR | string |
MONEY | string |
NUMERIC | decimal |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38 |
NUMRANGE | string |
PATH | string |
POINT | string |
POLYGON | string |
SERIAL | integer |
SMALLSERIAL | short |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | timestamp_ntz |
TIMESTAMP(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
TIMETZ | string |
TIMETZ(p) | string |
TSQUERY | string |
TSRANGE | string |
TSTZRANGE | string |
TSVECTOR | string |
TXID_SNAPSHOT | string |
UUID | string |
VARBIT | binary |
VARBIT(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Snowflake
Source data type | Spark data type |
|---|---|
ARRAY | string |
BINARY | binary |
BINARY(p) | binary |
BOOLEAN | boolean |
DATE | date |
DOUBLE | double |
GEOGRAPHY | string |
GEOMETRY | string |
NUMBER | decimal |
NUMBER(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMBER(p,s) | string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
OBJECT | string |
TIME | string |
TIME(p) | string |
TIMESTAMPLTZ | string |
TIMESTAMPLTZ(p) | string |
TIMESTAMPNTZ | timestamp_ntz |
TIMESTAMPNTZ(p) | timestamp_ntz |
TIMESTAMPTZ | timestamp |
TIMESTAMPTZ(p) | timestamp |
VARCHAR | varchar(p) |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
VARIANT | string |
Note: The Iceberg-to-Spark data type mapping used when data is read back from Iceberg with a Spark engine is different from the Spark-to-Iceberg mapping used when S3 Tables Writer writes to Iceberg. Do not assume the two mappings are symmetrical.
SQL Server
Source data type | Spark data type |
|---|---|
BIGINT | long |
BIGINT IDENTITY(p,s) | long, if 10 <= (p) <= 19 |
BINARY(p) | binary |
BIT | boolean |
CHAR | char(p) |
CHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
DATE | date |
DATETIME | timestamp_ntz |
DATETIME2 | timestamp_ntz |
DATETIME2(p) | timestamp_ntz |
DATETIMEOFFSET | timestamp |
DATETIMEOFFSET(p) | timestamp |
DECIMAL | decimal |
DECIMAL(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
DECIMAL(p,s) | string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
FLOAT | double |
FLOAT(p) | double, if (p) > 10 |
GEOGRAPHY | Not supported |
GEOMETRY | Not supported |
HIERARCHYID | Not supported |
IMAGE | binary |
INT | integer |
INT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 long, if 10 <= (p) <= 19 |
MONEY | string |
NCHAR | char(p) |
NCHAR(p) | string, if (p) > 2147483647 char(p), if (p) <= 2147483647 |
NTEXT | string |
NUMERIC | decimal |
NUMERIC IDENTITY(p,s) | string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,0) | decimal, if (p) <= 38, if (s) <= 38 |
NUMERIC(p,s) | string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38 |
NVARCHAR | varchar(p) |
NVARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
NVARCHAR(p) | varchar(p), if (p) <= 2147483647 |
REAL | double |
REAL(p) | double |
SMALLDATETIME | timestamp_ntz |
SMALLINT | short |
SMALLINT IDENTITY(p,s) | integer, if 5 <= (p) <= 10 short, if 3 <= (p) <= 5 |
SMALLMONEY | string |
SQL_VARIANT | Not supported |
TEXT | string |
TIME | string |
TIME(p) | string |
TIMESTAMP | binary |
TINYINT | byte |
TINYINT IDENTITY(p,s) | byte, if (p) <= 3 short, if 3 <= (p) <= 5 |
UNIQUEIDENTIFIER | varchar(p) |
VARBINARY | binary |
VARBINARY(max) | binary |
VARBINARY(p) | binary |
VARCHAR | varchar(p) |
VARCHAR(max) | varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647 |
VARCHAR(p) | string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647 |
XML | string |
Writing snapshot and CDC data
Writing initial snapshot data
Run S3 Tables Writer in APPENDONLY mode for snapshot data. Snapshot loads work from RDBMS sources as well as other sources that support initial-load mode (for example SalesforceReader and MongoDBReader). Batches that meet the configured upload policy are executed as insert statements against the Iceberg tables.
Initial load parallelism
Multiple threads group operations from multiple tables and insert into Iceberg tables in parallel, accelerating initial loads. Configure the ParallelThreads property on both the source adapter (for example DatabaseReader) and S3 Tables Writer to enable it.
Initial load recovery
S3 Tables Writer supports initial-load recovery at table-level granularity: after a restart, a table that had already completed loading is not rewritten.
Writing CDC data
Run S3 Tables Writer in MERGE mode for CDC, with the source adapter capturing data in incremental/CDC mode. Every insert, update, and delete captured from the source is applied against the target Iceberg tables in the configured batches.
Optimized merge
With OptimizedMerge enabled, S3 Tables Writer performs CDC merges using only the updated field values and key fields for update and delete operations, letting source CDC readers send partial row images instead of full-row images. This significantly reduces the data volume processed for update-heavy and delete-heavy CDC workloads. OptimizedMerge is valid only in MERGE mode (see Configuration validation).
Upload policy and table mapping
Upload policy
Upload policy affects performance. A low event count creates many small batches, adding overhead; a very high event count creates large batch files with delayed upload and execution. Set the event count based on your data traffic — favor a higher value when traffic is high.
Upload policy parameters
A batch is uploaded once either the eventcount or interval threshold is reached.
Parameter | Description | Valid range | Default |
|---|---|---|---|
eventcount | Number of events buffered before uploading. | A positive number | 100000 |
interval | Maximum time events are buffered before uploading, in seconds (s, default) or minutes (m). | A positive number | 60s |
Table mapping
Table mapping is provided as source table,target table.
Multiple mappings are separated with semicolons (;).
The wildcard % matches multiple tables within a namespace.
Warning: S3 Tables requires all table, namespace, and column names to be lowercase. A wildcard mapping (for example SCOTT.%) against a source that uses uppercase identifiers by default, such as Oracle, will attempt to create uppercase-named target tables and fail. Use explicit, static, lowercase-mapped table names instead of wildcards when the source uses uppercase identifiers. See Limitations.
Key column support
S3 Tables Writer supports the keycolumns option for row identification in target tables, for example Tables:'<SCHEMA>.<TABLE>,<namespace>.<table> keycolumns(<key_column>)'.
Wildcards can be used for the source table provided all matched tables share the same key columns: Tables:'<SCHEMA>.%,<namespace>.% KeyColumns(...)'.
If keycolumns is not provided, the source table's primary key is used; if neither exists, Striim concatenates all column values to form the row identifier.
Column mapping support
Column mapping maps columns between source and target when the structures don't directly match, in order (source to target). Use it to reorder columns or to skip unmapped source columns.
Syntax: <source table>,<target table> ColumnMap(<target column>=<source column>,...)
DDL support and special character support
DDL support
S3 Tables Writer executes certain DDL changes transmitted from source adapters against the S3 Tables Iceberg warehouse. Every operation type the source can send has a specific, stated result — S3 Tables Writer does not treat unlisted DDL operations the same as any of the operations below.
Supported source DDL operation | S3 Tables Writer behavior |
|---|---|
CREATE SCHEMA | Creates the Iceberg namespace. |
CREATE TABLE | Processes the statement and creates an Iceberg table at the target. On the AWS Glue catalog path, this requires the AWS-supplied EMR patch described in Limitations — contact Striim Support for guidance. |
ALTER TABLE ... ADD COLUMN | Processes the statement and adds a column to the target table. |
ALTER TABLE ... MODIFY COLUMN | Processes the statement and modifies the column on the target table. |
ALTER TABLE ... ADD/DROP PRIMARY KEY, or ADD/DROP CONSTRAINT ... PRIMARY KEY | Iceberg has no PRIMARY KEY or CONSTRAINT concept. S3 Tables Writer skips this DDL operation from the source and continues processing; it does not generate any target DDL and does not halt the application. |
ALTER TABLE ... DROP COLUMN | Processes the statement and drops the column from the target table. |
DROP TABLE | Processes the statement and drops the table. |
TRUNCATE TABLE | Processes the statement and truncates the table. |
Special character support
The only special character supported in a table name is the underscore (_). This applies regardless of write mode (APPENDONLY or MERGE) — table and namespace names are a property of the S3 Tables resource itself, fixed at creation. All other special characters ($ @ # % ^ & * ( ) \ / + = ? [ ] { } - ! |) are not supported in table names. Column-name support for special characters is not confirmed for this release; use underscores in column names, consistent with table names, until this is confirmed.
Connection retry support
Connection retry is configured independently at each component's own connection profile, using that profile's ConnectionRetryPolicy property:
Compute engine (AWS EMR): ConnectionRetryPolicy on the AWS EMR connection profile.
External stage (S3): ConnectionRetryPolicy on the S3 connection profile.
S3 Tables: ConnectionRetryPolicy on the S3Tables connection profile.
Note: AWS enforces region-level API rate limits on S3 Tables and related services. Striim retries throttled S3 Tables API calls automatically using an internal exponential back-off policy. Under high data traffic, or when running multiple S3 Tables Writer applications against the same AWS account and region simultaneously, the default AWS API quota may not be sufficient — retries can be exhausted, causing throttling and added latency. This is a constraint of the AWS platform, not a Striim defect. Monitor for throttling errors and request an AWS API quota increase proportional to the number of S3 Tables Writer applications you run and the data volume processed.
Required client libraries
S3 Tables Writer uses the AWS SDK for Amazon EMR to submit and monitor jobs on the compute engine, and the AWS SDK for Amazon S3 to read and write the external staging area. Both are standard AWS SDK components; no additional client-side installation is required beyond what Striim ships with the adapter.