Skip to main content

Create an S3 Tables Writer application

Creating an S3 Tables Writer application involves provisioning AWS infrastructure (an S3 Tables bucket, a general-purpose S3 staging bucket, an Amazon EMR cluster, and — for the AWS Glue catalog path — AWS Glue and Lake Formation permissions), creating the corresponding Striim connection profiles, and then building a pipeline that uses S3TablesIcebergWriter as the target. You can build the pipeline using the Flow Designer or TQL.

Prerequisites for creating an S3 Tables Writer application

S3 Tables Writer supports two catalog paths, selected by the writer's CatalogType property: the S3 Tables embedded catalog (default) or the AWS Glue Catalog. The AWS-side prerequisites differ between the two paths.

S3 Tables catalog path

  • An AWS EMR cluster (release emr-7.13.0 or later, with the Spark and Hadoop applications and the iceberg-defaults classification enabled), with a corresponding AWS EMR connection profile in Striim.

  • An S3 Tables bucket to store the Iceberg tables, with a corresponding S3Tables connection profile in Striim.

  • A general-purpose S3 bucket location, reachable from the EMR cluster, to use as the external stage, with a corresponding S3 connection profile in Striim.

  • An IAM role attached to the EMR cluster's instance profile, granted the S3 Tables permissions the compute engine needs (see S3 Tables Writer initial setup).

AWS Glue catalog path

In addition to the S3 Tables catalog path prerequisites above:

  • The S3 Tables bucket must have Glue integration enabled (Enable integration, in the S3 Table buckets console) before it is created.

  • An AWS Glue Catalog connection profile in Striim, in addition to the S3Tables connection profile (S3Tables remains the data lake connection profile even when Glue is the catalog).

  • The EMR instance-profile IAM role must additionally be granted the required AWS Glue permissions.

  • Lake Formation permissions granting that same IAM role access to the Glue database (namespace) that S3 Tables Writer writes to — configured under Lake Formation, not IAM.

Warning: If your pipeline issues CREATE TABLE DDL against the AWS Glue catalog, an additional AWS-supplied patch must be applied to the EMR cluster; without it, CREATE TABLE fails on EMR. Contact Striim Support for guidance before relying on CREATE TABLE DDL propagation with the Glue catalog path. See S3 Tables Writer operational considerations, Limitations.

Full step-by-step instructions for both paths are in S3 Tables Writer initial setup.

Create an S3 Tables Writer application using a wizard

When you create an S3 Tables Writer target using a wizard, you specify properties equivalent to the following, based on the wizard for the existing Iceberg Writer adapter:

  • Data Lake Connection Profile Name: the S3Tables connection profile you created (see Prerequisites for creating an S3 Tables Writer application).

  • Compute Connection Profile Name: the AWS EMR connection profile you created.

  • Catalog Connection Profile Name: required only for the AWS Glue catalog path — the AWS Glue Catalog connection profile you created.

  • External Staging Location: the S3 bucket location that will hold staging files.

  • Iceberg Tables Location: the S3 Tables bucket ARN that will hold the Iceberg tables.

  • Write mode: select Merge for continuous CDC replication or Append Only for snapshot loads.

Striim creates namespaces and tables in S3 Tables corresponding to the source schemas and tables you select in the wizard.

Placeholder: the field names and steps above are carried over from the equivalent Iceberg Writer wizard and need to be confirmed and replaced with the actual S3 Tables Writer wizard fields before this topic is published.

Create an S3 Tables Writer application using the Flow Designer

Before you begin: complete the prerequisites for your chosen catalog path and create the required connection profiles (see S3 Tables Writer initial setup).

  1. In the Flow Designer, add a source component for your source (for example, Database Reader configured for Oracle, SQL Server, MySQL, or PostgreSQL) and configure it.

  2. In the Components search bar, under Target Components, type S3Tables to locate S3TablesIcebergWriter, then drag it onto the canvas.

  3. Connect the source component's output stream to the S3TablesIcebergWriter component's input.

  4. Configure the target's properties: enter DataLakeConnectionProfileName, ComputeConnectionProfileName, and ExternalStageConnectionProfileName using the connection profiles you created; enter ExternalStagingLocation; select Mode; and enter Tables to map source tables to target tables.

  5. If you are using the AWS Glue catalog path, set CatalogType to AWSGlueCatalog and enter CatalogConnectionProfileName.

  6. Save the application.

Result: the application appears in Created state and is ready to deploy.

Next step: deploy and run the application, then monitor it using the metrics in S3 Tables Writer monitoring metrics.

Create an S3 Tables Writer application using TQL

You can create an S3 Tables Writer application entirely in TQL using the Striim Console. The examples below use the S3 Tables catalog path and the AWS Glue catalog path respectively; replace the placeholder connection profile names, table mappings, and paths with your own values.

Example 1: S3 Tables catalog

CREATE OR REPLACE TARGET WriteToS3Table USING Global.S3TablesIcebergWriter (
  optimizedMerge: false,
  UploadPolicy: 'eventcount:100000,interval:6s',
  Tables: '<source-schema>.<SOURCE_TABLE>,<target-namespace>.<target_table>',
  CatalogConnectionProfileName: '',
  CDDLAction: 'Process',
  ComputeConnectionProfileName: 'admin.EMRCP',
  ConnectionRetryPolicy: 'initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=60s, maxAttempts=10, totalTimeout=10m, jitterEnabled=true',
  DataLakeConnectionProfileName: 'admin.S3Tables',
  CatalogType: 'S3Tables',
  Mode: 'APPENDONLY',
  ExternalStagingLocation: '<staging-bucket>/stage',
  ExternalStageConnectionProfileName: 'admin.S3Stage' )
INPUT FROM SourceOutputStream;

CatalogConnectionProfileName is left empty because the S3 Tables catalog path uses the embedded catalog and requires no separate catalog connection profile.

Example 2: AWS Glue catalog

CREATE OR REPLACE TARGET WriteToS3TablesTarget USING Global.S3TablesIcebergWriter (
  DataLakeConnectionProfileName: 'admin.S3TablesCP',
  ExternalStagingLocation: '<staging-bucket>/stage',
  UploadPolicy: 'eventcount:100000,interval:10s',
  optimizedMerge: false,
  CatalogConnectionProfileName: 'admin.AWSGlueCP',
  ExternalStageConnectionProfileName: 'admin.S3StageCP',
  ComputeConnectionProfileName: 'admin.AWSEMRCP',
  Mode: 'MERGE',
  CDDLAction: 'Process',
  CatalogType: 'AWSGlueCatalog',
  Tables: '<schema>.<TABLE1>,<namespace>.<table1>;<schema>.<TABLE2>,<namespace>.<table2>' )
INPUT FROM SourceCDCOutputStream;

Both examples set CDDLAction to Process so that supported source DDL is propagated automatically; see DDL support in S3 Tables Writer programmer's reference for what is and isn't supported per catalog.

Getting started: Oracle to S3 Tables Writer tutorial

This tutorial builds a complete Initial Load (snapshot) application and a companion CDC application that move data from an Oracle source into S3 Tables, using the Oracle Reader source wizards together with manual configuration of the S3 Tables Writer target. It covers both the S3 Tables catalog path and the AWS Glue catalog path; follow whichever matches your target catalog. Oracle-side configuration (ARCHIVELOG mode, supplemental logging, and required grants) is out of scope here — see the Oracle Reader documentation for source-side setup.

Example 1: build and run the Initial Load application

Before you begin: complete the AWS-side setup in S3 Tables Writer initial setup for your chosen catalog path, and create the AWS EMR, S3, S3Tables, and (Glue path only) AWS Glue Catalog connection profiles described there.

  1. Go to Apps, click Create an App, and choose Start from scratch.

  2. Drag Oracle Reader (Initial Load) from the source components onto the canvas and complete the source wizard: enter the Oracle connection details, then select the schemas and tables to migrate.

  3. When the wizard finishes, Striim creates the Oracle source component with a default output stream.

  4. Enable CreateSchema on the Oracle Reader source so that S3 Tables Writer can automatically create the target namespaces and tables (see Initial Schema Creation & Data Type Mapping in S3 Tables Writer programmer's reference).

  5. Save the source component.

  6. In the Components search bar, under Target Components, type S3Tables and drag S3TablesIcebergWriter onto the canvas.

  7. Configure the target:

    • Set the input stream to the Oracle source's output stream.

    • DataLakeConnectionProfileName: the S3Tables connection profile you created.

    • ComputeConnectionProfileName: the AWS EMR connection profile you created.

    • ExternalStageConnectionProfileName: the S3 connection profile you created.

    • ExternalStagingLocation: a folder path inside the staging bucket, for example <staging-bucket>/stage.

    • CatalogType: S3Tables (default), or AWSGlueCatalog. If AWSGlueCatalog, also set CatalogConnectionProfileName to the AWS Glue Catalog connection profile.

    • Mode: APPENDONLY, for the initial load.

    • Tables: the Oracle-to-target table mapping. Target table and namespace names must be lowercase — an uppercase-identifier Oracle source combined with a wildcard mapping (for example SCOTT.%) will fail. Use explicit, static, lowercase-mapped table names instead. See Limitations in S3 Tables Writer operational considerations.

  8. Save the application.

Deploy and run the Initial Load application

  1. Deploy the application.

  2. Run the application. Striim replicates the current snapshot of the selected Oracle tables into the S3 Tables (or Glue-cataloged) Iceberg tables.

Result: the selected Oracle tables are created and populated as Iceberg tables in S3 Tables.

Next step: monitor progress using S3 Tables Writer monitoring metrics, then build the companion CDC application in Example-2.

Example 2: build and run the CDC application

  1. Drag Oracle Reader (CDC) from the source components onto the canvas and complete the source wizard: enter the Oracle connection details, then select the same schemas and tables used for the initial load.

  2. Add and configure an S3TablesIcebergWriter target the same way as for the Initial Load application, with two differences:

    • Input stream: the output stream from the Oracle CDC source, not the Initial Load source.

    • Mode: MERGE, not APPENDONLY.

  3. Save the application.

Result: the CDC application is in Created state.

Deploy and run the CDC application

  1. Deploy the application.

  2. Run the application. Striim applies live CDC changes (inserts, updates, and deletes) from Oracle to the target Iceberg tables.

Monitoring

Use S3 Tables Writer monitoring metrics for table-level and adapter-level throughput and latency on both applications. If either application halts, start with the troubleshooting table in S3 Tables Writer operational considerations.