S3 Tables Writer programmer's reference

This reference describes every property of S3 Tables Writer and by its four connection profiles, plus the data type mapping, DDL support, and special-character rules that govern how source data is represented in S3 Tables.

S3 Tables Writer properties

property	type	default value	notes
Catalog Connection Profile Name	String		Required and appears in UI only when Catalog Type is AWS Glue. If the Catalog Type is S3 Tables, do not specify this property in TQL. See Create the connection profiles and Connection profiles.
Catalog Type	enum	S3 Tables	With the default value, S3 Tables Writer will use S3 as its catalog. Set to AWS Glue Catalog to use that instead. For more information, see S3 Tables Writer initial setup.
CDDL Action	String	Process	See Handling schema evolution..
Compute Connection Profile Name	String		See Create the connection profiles and Connection profiles.
Data Lake Connection Profile Name	String		See Create the connection profiles and Connection profiles.
Excluded Tables	String		If `Tables` uses a wildcard, data from any tables specified here will be omitted. Multiple table names (separated by semicolons) and wildcards may be used exactly as for `Tables`.
External Stage Connection Profile Name	String		See Create the connection profiles and Connection profiles.
External Staging Location	String		Path to a folder in the S3 bucket used as the staging area for batch data, metadata, and metrics files during execution. The data is cleared when the adapter closes. Path must start from the bucket name.
Mode	enum	APPENDONLY	With the default setting of APPENDONLY, update and delete operations in the source are handled as inserts in the target. Use this mode for initial loads, audit tables, or non-CDC sources. Set to MERGE to handle update and delete operations in the source as updates and deletes in the target., so the target replicates the source.
Optimized Merge	Boolean	False	Appears in Flow Designer only when Mode is Merge. Set to True only when Mode is MERGE and the target's input stream is the output of an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events will include partial records. For example, with Oracle Reader, when supplemental logging has not been enabled for all columns, partial records are sent for updates. When the source events will always include full records, leave this set to False. When Optimized Merge is True, each primary key update is handled in a separate write operation. If the source has frequent primary key updates, this may lead to a decline in write performance compared with Optimized Merge = False.
Parallel Threads	Integer	1	Not supported when Mode is Merge. See Creating multiple writer instances (parallel threads).
Tables	String	User supplied	See Table mapping.
Upload Policy	String	eventcount:100000,interval:60s	The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Cached data is written to the storage account every time any of the specified values is exceeded. With the default value, data will be written every five minutes or sooner if the cache contains 10,000 events. When the app is undeployed, all remaining data is written to the storage account.

Connection profiles

AWS EMR connection profile

Endpoint name: AWS EMR

Property	Type / default	Description	Alterable
AccessKey	String, optional	Optional AWS access key for authenticating with AWS services. If not provided, uses the default AWS credentials chain or IAM role-based authentication.	Yes
AdditionalConfiguration	String, optional	Additional Spark/Hadoop configuration properties in key=value format, applied to EMR jobs.	No
ClientConfiguration	String, default MaxErrorRetry=5	SDK-level retry settings.	No
ClusterID	String, required	The unique identifier of the AWS EMR cluster used as the compute engine for Iceberg operations.	No
JarLocations	String, optional	Custom JAR file locations (S3 or local paths) to include in Spark job execution. Multiple locations can be provided.	No
Region	String, required	The AWS region where the EMR cluster is deployed (for example, us-east-1).	No
SecretKey	Password, optional	Optional AWS secret key corresponding to the access key, stored securely. If not provided, uses the default AWS credentials chain or IAM role-based authentication.	Yes

S3 Tables connection profile

Endpoint name: S3 Tables

Property	Type / default	Description	Alterable
AdditionalConfiguration	String, optional	Additional S3 Tables-specific configuration properties in key=value format, applied when connecting to the bucket.	No
Region	String, required	The AWS region where the S3 Tables bucket is deployed.	No
S3TablesBucketARN	String, required	The full ARN of the S3 Tables bucket used as the data lake. Format: arn:aws:s3tables:region:account-id:bucket/bucket-name.	No

Note: S3TablesBucketARN can be found in the S3 Tables bucket's properties in the AWS Console. Authentication to S3 Tables happens exclusively through the IAM role attached to the EMR cluster's instance profile. Access Key and Secret Key are not exposed as configurable properties on this connection profile. This is intentional design, not a defect.

AWS Glue Catalog connection profile

Endpoint name: AWS Glue Catalog

Property	Type / default	Description	Alterable
AccountID	String, required	The 12-digit AWS account ID that owns the Glue Catalog.	No
AdditionalConfiguration	String, optional	Additional Glue Catalog configuration properties in key=value format (for example, catalog-encryption-mode=DISABLED).	No
region	String, required, lowercase	The AWS region where the Glue Catalog is deployed.	No

Note: Authentication to Glue happens exclusively through the same EMR instance-profile IAM role used for S3 Tables, which must additionally be granted the Glue-specific permissions and the Lake Formation grant described in S3 Tables Writer initial setup. Access Key and Secret Key are not exposed as configurable properties on this connection profile.

S3 connection profile (for external staging)

Endpoint name: S3

Property	Type / default	Description	Alterable
s3AccessKey	Password, optional	Optional AWS access key for S3 authentication, stored securely. If not provided, uses the default AWS credentials chain or IAM role-based authentication.	Yes
s3BucketName	String, default striim-bucket — not functional	Not used to resolve the external stage bucket — the bucket is always derived from the writer's ExternalStagingLocation property instead. Planned for removal.	No
s3Region	String, default us-west-1	The AWS region where the S3 staging bucket is deployed.	No
s3SecretAccessKey	Password, optional	Optional AWS secret key corresponding to the access key, stored securely.	Yes

Note: If s3AccessKey and s3SecretAccessKey are not provided, the adapter uses IAM credentials configured in the environment. Altering s3AccessKey or s3SecretAccessKey on a running application takes effect on the next connection retry.

Initial schema creation and data type mapping

S3 Tables Writer creates the corresponding namespaces and tables automatically during initial load if they do not already exist.

Enable CreateSchema on the source adapter (for example DatabaseReader) when configuring an initial load pipeline. Auto schema creation is available from any RDBMS source that connects through DatabaseReader over JDBC, including Oracle, SQL Server, MySQL, and PostgreSQL.

During the snapshot load, DDL statements for creating namespaces, followed by tables, are sent to S3 Tables Writer, which executes them one after another on the compute engine. Once all DDL statements are executed, the snapshot data follows.

Schema Conversion Utility

The Schema Conversion command-line utility generates the initial target schema from source connection details, without running a full Striim application:

bin/schemaConversionUtility.sh -s=oracle -d="jdbc:oracle:thin:@//<host>:<port>/<sid>" -u="<username>" -p="<password>" -b='<schema>.<TABLE>;' -t="spark"

Parameter	Description
-s	Source database type.
-d	JDBC URL to the source database.
-u	Username.
-p	Password.
-b	Semicolon-separated list of source tables to generate a schema for.
-t	Target type: spark.

The utility returns SQL files containing target-compatible DDL for creating S3 Tables, which you can execute directly with any Spark engine or the AWS Athena query console.

Data type mapping

S3 Tables Writer maps each source column's data type to a Spark data type when writing to Iceberg. The following tables list, for each supported source, which source data types map to each Spark data type. Where a mapping depends on the column's declared precision or scale, the condition is shown in parentheses.

BigQuery

Source data type	Spark data type
BIGNUMERIC	string
BIGNUMERIC(p,0)	decimal, if (p) <= 38, if (s) <= 38
BIGNUMERIC(p,s)	string decimal, if (p) <= 38, if (s) <= 38
BOOL	boolean
BYTES	binary
BYTES(p)	binary
DATE	date
DATETIME	timestamp_ntz
FLOAT64	double
GEOGRAPHY	varchar(p)
INT64	long
INTERVAL	varchar(p)
JSON	string
NUMERIC	decimal
NUMERIC(p,0)	decimal, if (p) <= 38, if (s) <= 38
NUMERIC(p,s)	decimal, if (p) <= 38, if (s) <= 38
STRING	varchar(p)
TIME	string
TIMESTAMP	timestamp

MySQL

Source data type	Spark data type
BIGINT UNSIGNED	Not supported
BINARY	binary
BINARY(p)	binary
BIT	binary
BIT(p)	binary
BLOB	binary
CHAR	char(p)
CHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
DATE	date
DATETIME	timestamp_ntz
DATETIME(s)	timestamp_ntz
DECIMAL	decimal
DECIMAL UNSIGNED	decimal
DECIMAL UNSIGNED(p,0)	decimal, if (p) <= 38, if (s) <= 38
DECIMAL UNSIGNED(p,s)	string, if (p,s) > 38, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
DECIMAL(p,0)	decimal, if (p) <= 38, if (s) <= 38
DECIMAL(p,s)	string, if (p,s) > 38, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
DOUBLE	double
DOUBLE(p,0)	double
DOUBLE(p,s)	double
ENUM	string
FLOAT	double
FLOAT(p,s)	double, if (p,s) > 10
GEOMETRY	Not supported
INT	integer
INT UNSIGNED	integer
JSON	string
LONGBLOB	binary
LONGTEXT	string
MEDIUMBLOB	binary
MEDIUMINT	integer
MEDIUMINT UNSIGNED	integer
MEDIUMTEXT	string
SET	Not supported
SMALLINT	short
SMALLINT UNSIGNED	short
TEXT	string
TIME	string
TIMESTAMP	timestamp_ntz
TINYBLOB	binary
TINYINT	byte
TINYINT UNSIGNED	byte
TINYTEXT	string
VARBINARY(p)	binary
VARCHAR(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
YEAR	integer

Oracle

Source data type	Spark data type
BFILE	Not supported
BINARY_DOUBLE	double
BINARY_FLOAT	float
BLOB	binary
CHAR	char(p)
CHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
CLOB	string
FLOAT	double
FLOAT(p)	float, if (p) <= 10 double, if (p) > 10
LONG	Not supported
NCHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
NCLOB	string
NUMBER	decimal
NUMBER(p,0)	decimal, if (p) <= 38, if (s) <= 38
NUMBER(p,s)	decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38
NVARCHAR2(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
RAW(p)	binary
ROWID	string
SDO_GEOMETRY	Not supported
UROWID	Not supported
VARCHAR2(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
XMLTYPE	string

PostgreSQL

Source data type	Spark data type
BIGSERIAL	long
BIT	binary
BIT(p)	binary
BOOL	boolean
BOX	string
BPCHAR	char(p)
BPCHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
BYTEA	binary
CIDR	string
CIRCLE	string
DATE	date
DATERANGE	string
FLOAT4	float
FLOAT8	double
INET	string
INT2	short
INT4	integer
INT4RANGE	string
INT8	long
INT8RANGE	string
INTERVAL	string
INTERVAL(p)	string
JSON	string
JSONB	string
LINE	string
LSEG	string
MACADDR	string
MONEY	string
NUMERIC	decimal
NUMERIC(p,0)	decimal, if (p) <= 38, if (s) <= 38
NUMERIC(p,s)	decimal, if (p) <= 38, if (s) <= 38 string, if (p,s) > 38, if (s) > 38
NUMRANGE	string
PATH	string
POINT	string
POLYGON	string
SERIAL	integer
SMALLSERIAL	short
TEXT	string
TIME	string
TIME(p)	string
TIMESTAMP	timestamp_ntz
TIMESTAMP(p)	timestamp_ntz
TIMESTAMPTZ	timestamp
TIMESTAMPTZ(p)	timestamp
TIMETZ	string
TIMETZ(p)	string
TSQUERY	string
TSRANGE	string
TSTZRANGE	string
TSVECTOR	string
TXID_SNAPSHOT	string
UUID	string
VARBIT	binary
VARBIT(p)	binary
VARCHAR	varchar(p)
VARCHAR(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
XML	string

Snowflake

Source data type	Spark data type
ARRAY	string
BINARY	binary
BINARY(p)	binary
BOOLEAN	boolean
DATE	date
DOUBLE	double
GEOGRAPHY	string
GEOMETRY	string
NUMBER	decimal
NUMBER(p,0)	decimal, if (p) <= 38, if (s) <= 38
NUMBER(p,s)	string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
OBJECT	string
TIME	string
TIME(p)	string
TIMESTAMPLTZ	string
TIMESTAMPLTZ(p)	string
TIMESTAMPNTZ	timestamp_ntz
TIMESTAMPNTZ(p)	timestamp_ntz
TIMESTAMPTZ	timestamp
TIMESTAMPTZ(p)	timestamp
VARCHAR	varchar(p)
VARCHAR(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
VARIANT	string

Note: The Iceberg-to-Spark data type mapping used when data is read back from Iceberg with a Spark engine is different from the Spark-to-Iceberg mapping used when S3 Tables Writer writes to Iceberg. Do not assume the two mappings are symmetrical.

SQL Server

Source data type	Spark data type
BIGINT	long
BIGINT IDENTITY(p,s)	long, if 10 <= (p) <= 19
BINARY(p)	binary
BIT	boolean
CHAR	char(p)
CHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
DATE	date
DATETIME	timestamp_ntz
DATETIME2	timestamp_ntz
DATETIME2(p)	timestamp_ntz
DATETIMEOFFSET	timestamp
DATETIMEOFFSET(p)	timestamp
DECIMAL	decimal
DECIMAL(p,0)	decimal, if (p) <= 38, if (s) <= 38
DECIMAL(p,s)	string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
FLOAT	double
FLOAT(p)	double, if (p) > 10
GEOGRAPHY	Not supported
GEOMETRY	Not supported
HIERARCHYID	Not supported
IMAGE	binary
INT	integer
INT IDENTITY(p,s)	integer, if 5 <= (p) <= 10 long, if 10 <= (p) <= 19
MONEY	string
NCHAR	char(p)
NCHAR(p)	string, if (p) > 2147483647 char(p), if (p) <= 2147483647
NTEXT	string
NUMERIC	decimal
NUMERIC IDENTITY(p,s)	string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
NUMERIC(p,0)	decimal, if (p) <= 38, if (s) <= 38
NUMERIC(p,s)	string, if (p,s) > 38 string, if (s) > 38 decimal, if (p) <= 38, if (s) <= 38
NVARCHAR	varchar(p)
NVARCHAR(max)	varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647
NVARCHAR(p)	varchar(p), if (p) <= 2147483647
REAL	double
REAL(p)	double
SMALLDATETIME	timestamp_ntz
SMALLINT	short
SMALLINT IDENTITY(p,s)	integer, if 5 <= (p) <= 10 short, if 3 <= (p) <= 5
SMALLMONEY	string
SQL_VARIANT	Not supported
TEXT	string
TIME	string
TIME(p)	string
TIMESTAMP	binary
TINYINT	byte
TINYINT IDENTITY(p,s)	byte, if (p) <= 3 short, if 3 <= (p) <= 5
UNIQUEIDENTIFIER	varchar(p)
VARBINARY	binary
VARBINARY(max)	binary
VARBINARY(p)	binary
VARCHAR	varchar(p)
VARCHAR(max)	varchar(p), if (p) <= 2147483647 string, if (max) > 2147483647
VARCHAR(p)	string, if (p) > 2147483647 varchar(p), if (p) <= 2147483647
XML	string

Writing snapshot and CDC data

Writing initial snapshot data

Run S3 Tables Writer in APPENDONLY mode for snapshot data. Snapshot loads work from RDBMS sources as well as other sources that support initial-load mode (for example SalesforceReader and MongoDBReader). Batches that meet the configured upload policy are executed as insert statements against the Iceberg tables.

Initial load parallelism

Multiple threads group operations from multiple tables and insert into Iceberg tables in parallel, accelerating initial loads. Configure the ParallelThreads property on both the source adapter (for example DatabaseReader) and S3 Tables Writer to enable it.

Initial load recovery

S3 Tables Writer supports initial-load recovery at table-level granularity: after a restart, a table that had already completed loading is not rewritten.

Writing CDC data

Run S3 Tables Writer in MERGE mode for CDC, with the source adapter capturing data in incremental/CDC mode. Every insert, update, and delete captured from the source is applied against the target Iceberg tables in the configured batches.

Optimized merge

With OptimizedMerge enabled, S3 Tables Writer performs CDC merges using only the updated field values and key fields for update and delete operations, letting source CDC readers send partial row images instead of full-row images. This significantly reduces the data volume processed for update-heavy and delete-heavy CDC workloads. OptimizedMerge is valid only in MERGE mode (see Configuration validation).

Upload policy and table mapping

Upload policy

Upload policy affects performance. A low event count creates many small batches, adding overhead; a very high event count creates large batch files with delayed upload and execution. Set the event count based on your data traffic — favor a higher value when traffic is high.

Upload policy parameters

A batch is uploaded once either the eventcount or interval threshold is reached.

Parameter	Description	Valid range	Default
eventcount	Number of events buffered before uploading.	A positive number	100000
interval	Maximum time events are buffered before uploading, in seconds (s, default) or minutes (m).	A positive number	60s

Table mapping

Table mapping is provided as source table,target table.
Multiple mappings are separated with semicolons (;).
The wildcard % matches multiple tables within a namespace.

Warning: S3 Tables requires all table, namespace, and column names to be lowercase. A wildcard mapping (for example SCOTT.%) against a source that uses uppercase identifiers by default, such as Oracle, will attempt to create uppercase-named target tables and fail. Use explicit, static, lowercase-mapped table names instead of wildcards when the source uses uppercase identifiers. See Limitations.

Key column support

S3 Tables Writer supports the keycolumns option for row identification in target tables, for example Tables:'<SCHEMA>.<TABLE>,<namespace>.<table> keycolumns(<key_column>)'.

Wildcards can be used for the source table provided all matched tables share the same key columns: Tables:'<SCHEMA>.%,<namespace>.% KeyColumns(...)'.

If keycolumns is not provided, the source table's primary key is used; if neither exists, Striim concatenates all column values to form the row identifier.

Column mapping support

Column mapping maps columns between source and target when the structures don't directly match, in order (source to target). Use it to reorder columns or to skip unmapped source columns.

Syntax: <source table>,<target table> ColumnMap(<target column>=<source column>,...)

DDL support and special character support

DDL support

S3 Tables Writer executes certain DDL changes transmitted from source adapters against the S3 Tables Iceberg warehouse. Every operation type the source can send has a specific, stated result — S3 Tables Writer does not treat unlisted DDL operations the same as any of the operations below.

Supported source DDL operation	S3 Tables Writer behavior
CREATE SCHEMA	Creates the Iceberg namespace.
CREATE TABLE	Processes the statement and creates an Iceberg table at the target. On the AWS Glue catalog path, this requires the AWS-supplied EMR patch described in Limitations — contact Striim Support for guidance.
ALTER TABLE ... ADD COLUMN	Processes the statement and adds a column to the target table.
ALTER TABLE ... MODIFY COLUMN	Processes the statement and modifies the column on the target table.
ALTER TABLE ... ADD/DROP PRIMARY KEY, or ADD/DROP CONSTRAINT ... PRIMARY KEY	Iceberg has no PRIMARY KEY or CONSTRAINT concept. S3 Tables Writer skips this DDL operation from the source and continues processing; it does not generate any target DDL and does not halt the application.
ALTER TABLE ... DROP COLUMN	Processes the statement and drops the column from the target table.
DROP TABLE	Processes the statement and drops the table.
TRUNCATE TABLE	Processes the statement and truncates the table.

Special character support

The only special character supported in a table name is the underscore (_). This applies regardless of write mode (APPENDONLY or MERGE) — table and namespace names are a property of the S3 Tables resource itself, fixed at creation. All other special characters ($ @ # % ^ & * ( ) \ / + = ? [ ] { } - ! |) are not supported in table names. Column-name support for special characters is not confirmed for this release; use underscores in column names, consistent with table names, until this is confirmed.

Connection retry support

Connection retry is configured independently at each component's own connection profile, using that profile's ConnectionRetryPolicy property:

Compute engine (AWS EMR): ConnectionRetryPolicy on the AWS EMR connection profile.
External stage (S3): ConnectionRetryPolicy on the S3 connection profile.
S3 Tables: ConnectionRetryPolicy on the S3Tables connection profile.

Note: AWS enforces region-level API rate limits on S3 Tables and related services. Striim retries throttled S3 Tables API calls automatically using an internal exponential back-off policy. Under high data traffic, or when running multiple S3 Tables Writer applications against the same AWS account and region simultaneously, the default AWS API quota may not be sufficient — retries can be exhausted, causing throttling and added latency. This is a constraint of the AWS platform, not a Striim defect. Monitor for throttling errors and request an AWS API quota increase proportional to the number of S3 Tables Writer applications you run and the data volume processed.

Required client libraries

S3 Tables Writer uses the AWS SDK for Amazon EMR to submit and monitor jobs on the compute engine, and the AWS SDK for Amazon S3 to read and write the external staging area. Both are standard AWS SDK components; no additional client-side installation is required beyond what Striim ships with the adapter.