S3 Tables Writer initial setup

Setting up S3 Tables Writer involves two phases: provisioning the required AWS infrastructure (this topic's first subtopic), and creating the corresponding Striim connection profiles and validating your configuration (Configure Striim to work with S3 Tables). Complete the AWS-side setup for your chosen catalog path before creating connection profiles in Striim.

Initial setup for S3 Tables Writer in AWS

The steps below cover the S3 Tables catalog path first, followed by the additional and different steps required for the AWS Glue catalog path.

S3 Tables catalog path

Step 1: Create the S3 Table bucket

Log in to the Amazon S3 Console using an account with administrator permissions.
In the left navigation pane, select Table buckets.
Click Create table bucket.
Enter a table bucket name, for example <table-bucket-name>.
Click Create table bucket.

Step 2: Create the external staging S3 bucket

Log in to the Amazon S3 Console.
Select General purpose buckets.
Click Create bucket.
Enter a bucket name, for example <staging-bucket>.
Click Create bucket.

Note: The external staging bucket is separate from the S3 Table bucket, and must be reachable from the EMR cluster.

Step 3: Create the EMR cluster

Open the Amazon EMR Console and click Create cluster.
Select the emr-7.13.0 release.
Select the Spark and Hadoop applications.
Under cluster scaling, choose the number of task and core nodes (one core node and one task node is sufficient for a single application).

Under Software settings, enter the following JSON configuration to enable Iceberg support:

{
  "Classification": "iceberg-defaults",
  "Properties": {
    "iceberg.enabled": "true"
  }
}

Under IAM, choose Create a service role for the EMR service role, and Create an instance profile for the EC2 instance profile role. Under S3 Bucket Access, choose Specific S3 buckets or prefixes and select the staging bucket created in Step 2.
Under Steps, enable Run multiple steps in parallel and set Maximum concurrent steps to at least the number of S3 Tables Writer instances that will use this cluster.
Leave the remaining settings at their defaults unless you have a specific requirement, then click Create cluster.

Cluster creation takes 5 to 10 minutes. Wait for the cluster to reach the WAITING state before continuing to Step 4.

Step 4: Create the IAM user or role for Striim

This identity lets Striim access the EMR cluster and the staging bucket to submit jobs and write data files.

In the IAM console, go to Users and click Create user (for example, striim-writer-user).
Choose Attach policies directly, then Create policy, then JSON, and enter the policy from Step 5, replacing the placeholders with your actual values.

Step 5: Create the IAM policy for EMR compute

In the IAM console, go to Policies and click Create policy, then JSON.

Enter the following JSON, replacing <region>, <account-id>, and <table-bucket-name> with your actual values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3TablesAccess",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTable",
        "s3tables:DeleteTable",
        "s3tables:CreateTable",
        "s3tables:GetNamespace",
        "s3tables:PutTableData",
        "s3tables:CreateNamespace",
        "s3tables:DeleteNamespace",
        "s3tables:ListTables",
        "s3tables:GetTableMetadataLocation",
        "s3tables:UpdateTableMetadataLocation",
        "s3tables:GetTableData",
        "s3tables:ListNamespaces",
        "s3tables:GetTableBucket"
      ],
      "Resource": [
        "arn:aws:s3tables:<region>:<account-id>:bucket/<table-bucket-name>/table/*",
        "arn:aws:s3tables:<region>:<account-id>:bucket/<table-bucket-name>"
      ]
    },
    {
      "Sid": "S3TablesListBuckets",
      "Effect": "Allow",
      "Action": "s3tables:ListTableBuckets",
      "Resource": "*"
    }
  ]
}

Give the policy a name, for example emr-s3tables-policy.

Step 6: Assign the IAM role to the EMR instance profile

Once the EMR cluster is in the WAITING state, open its Properties and note the EC2 instance profile value.
In the IAM console, go to Roles, find that role, click Add permissions then Attach policies, and attach the policy created in Step 5.

AWS Glue catalog path — additional and different steps

The AWS Glue catalog path follows the same six steps as the S3 Tables catalog path above, with these differences:

Step 1 (Glue variant): Enable Glue integration before creating the table bucket

In the Amazon S3 Console, under Table buckets, click Enable integration. Confirm that the integration is enabled for your table buckets.
Click Create table bucket, enter the table bucket name (for example <table-bucket-name>), and click Create table bucket.

Step 3 (Glue variant): EMR cluster and the CREATE TABLE patch

Create the EMR cluster the same way as Step 3 above (emr-7.13.0, Spark and Hadoop applications). If your pipeline needs to issue CREATE TABLE DDL against the Glue catalog, initial schema migration and CREATE TABLE DDL require an additional AWS-supplied patch applied to the EMR cluster. Contact Striim Support for guidance before relying on this capability.

Step 5 (Glue variant): Create the IAM policy for EMR compute with Glue and Lake Formation access

In the IAM console, go to Policies, click Create policy, then JSON.

Enter the following JSON, replacing <s3-tables-bucket>, <region>, and <account-id> with your actual values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueAccess",
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase", "glue:CreateTable", "glue:GetTables",
        "glue:UpdateTable", "glue:GetCatalog", "glue:GetDatabases", "glue:GetTable"
      ],
      "Resource": [
        "arn:aws:glue:<region>:<account-id>:table/s3tablescatalog/<s3-tables-bucket>/*",
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:catalog/s3tablescatalog",
        "arn:aws:glue:<region>:<account-id>:catalog/s3tablescatalog/<s3-tables-bucket>"
      ]
    },
    {
      "Sid": "LakeFormationAccess",
      "Effect": "Allow",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "*"
    },
    {
      "Sid": "S3TablesAccess",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTable", "s3tables:DeleteTable", "s3tables:CreateTable",
        "s3tables:GetNamespace", "s3tables:PutTableData", "s3tables:CreateNamespace",
        "s3tables:DeleteNamespace", "s3tables:ListTables", "s3tables:GetTableMetadataLocation",
        "s3tables:UpdateTableMetadataLocation", "s3tables:GetTableData", "s3tables:ListNamespaces",
        "s3tables:GetTableBucket"
      ],
      "Resource": [
        "arn:aws:s3tables:<region>:<account-id>:bucket/<s3-tables-bucket>/table/*",
        "arn:aws:s3tables:<region>:<account-id>:bucket/<s3-tables-bucket>"
      ]
    },
    {
      "Sid": "S3TablesListBuckets",
      "Effect": "Allow",
      "Action": "s3tables:ListTableBuckets",
      "Resource": "*"
    }
  ]
}

Give the policy a name (for example emr-s3tables-glue-policy), and attach it to the EMR instance-profile role the same way as Step 6 above.

Step 7 (Glue path only): Define access control in Lake Formation

Sign in as an administrator and go to Lake Formation.
Under Administration, select Application integration settings, enable Allow external engines to access data in Amazon S3 locations with full table access, and save.
Under Data permissions, click Grant.
Under Principals, select IAM users and roles, and select the EMR instance-profile role.
Under LF-Tags or catalog resources, select Named Data Catalog resources.
For Catalogs, select the subcatalog created when the table bucket was integrated with Glue, for example <account-id>:s3tablescatalog/<s3-tables-bucket>.
For Databases, select the namespace or namespaces that S3 Tables Writer will write to. These namespaces are already registered under the catalog selected above — you do not need to create a separate Glue database.
To grant broad access, leave Tables unselected and select Super under Database permissions. To scope access to a single table, select it under Tables and select Super under Table permissions.
Click Grant.

Configure Striim to work with S3 Tables

After the AWS infrastructure exists, create the Striim connection profiles that S3 Tables Writer uses, then verify your application configuration against the compile-time validation rules.

Create the connection profiles

S3 Tables Writer uses up to four connection profiles. Create them from Manage Striim, Connection Profiles, Create Connection Profile.

Connection profile (endpoint type)	Required for	Purpose
AWSEMR	Both catalog paths	The Amazon EMR cluster used as the compute engine.
AWS GlueC atalog	AWS Glue catalog path only	The AWS Glue Data Catalog used to track namespaces and tables. Leave CatalogConnectionProfileName empty and omit this profile on the S3 Tables catalog path.
S3	Both catalog paths	The general-purpose S3 bucket used as the external staging area.
S3 Tables	Both catalog paths	The S3 Tables bucket used as the data lake — always the DataLakeConnectionProfileName, even on the AWS Glue catalog path.

AWS EMR connection profile — select endpoint name AWSEMR. Enter ClusterID and Region for the cluster you created. Leave AccessKey and SecretKey blank to use the EMR instance-profile role (recommended), or supply an explicit key pair if you need Striim's own EMR calls to use different credentials than the instance profile.
S3 connection profile (external stage) — select endpoint name S3. Set s3Region to the region of the staging bucket. Leave s3AccessKey and s3SecretAccessKey blank to use the default AWS credentials chain, or supply them if the EMR instance profile does not have direct access to the staging bucket.
S3Tables connection profile — select endpoint name S3Tables. Set Region and S3TablesBucketARN to the S3 Tables bucket you created. On the S3 Tables catalog path, no separate catalog connection profile is needed — leave CatalogConnectionProfileName empty on the writer.
AWS Glue Catalog connection profile — Glue path only. Select endpoint name AWSGlueCatalog. Set region (lowercase) and AccountID to match the Glue catalog's account and region.

Note: The S3 connection profile's s3BucketName property is not functional — the bucket actually used for external staging always comes from the writer's ExternalStagingLocation property, not from any bucket-name field on the S3 connection profile. This field is planned for removal in a future release; do not rely on it.

Note: This alterability is specific to the S3 (external staging) connection profile; the S3Tables and AWS Glue Catalog connection profiles do not expose Access Key or Secret Key as configurable properties at all, because those two profiles authenticate solely through the EMR instance-profile IAM role.

Spark configuration: what Striim handles automatically compared to what you configure

S3 Tables Writer builds the Spark session configuration needed to reach the selected catalog automatically for every job; you do not configure Spark catalog settings directly. The customer-visible surface you do control is limited to the properties on the AWS EMR, S3Tables, and AWS Glue Catalog connection profiles — most notably each profile's AdditionalConfiguration property, which lets you pass extra key=value Spark or Hadoop properties (for example spark.executor.memory=4g) into the job, and each profile's Region (and, for the Glue path, AccountID), which S3 Tables Writer uses to build the catalog endpoint URI.

When CatalogType is S3Tables, Striim configures the Spark session to use the Iceberg REST catalog implementation with SigV4 request signing against the S3 Tables endpoint for the connection profile's region, and points the Spark warehouse setting at the S3 Tables bucket ARN from the S3Tables connection profile.
When CatalogType is AWSGlueCatalog, Striim configures the Spark session to use the Iceberg REST catalog implementation with SigV4 signing against the Glue endpoint for the connection profile's region, and points the Spark warehouse setting at the Glue subcatalog for your table bucket (account ID and table bucket name).

In both cases, Striim also enables Iceberg Spark extensions and case-sensitive SQL identifier handling. These settings are internal to how Striim submits the EMR job and are not user-configurable beyond the connection profile properties described above.

Configuration validation

When you compile an application containing S3 Tables Writer, Striim checks its properties before allowing the application to be deployed. The checks below run at compile time and surface as compiler errors — you must resolve them before the application will compile.

Required properties

The following properties must be non-blank: DataLakeConnectionProfileName, ComputeConnectionProfileName, ExternalStagingLocation, ExternalStageConnectionProfileName, and Tables.

Connection profile type matching

Each connection profile named in DataLakeConnectionProfileName, ComputeConnectionProfileName, and CatalogConnectionProfileName must be a connection profile of the matching type: an S3Tables profile for the data lake, an AWSEMR profile for compute, and (when CatalogType is AWSGlueCatalog) an AWSGlueCatalog profile for the catalog.

Condition	Compiler error
Named connection profile does not exist or cannot be looked up	Could not retrieve the connection profile '<name>' provided for <property>
Named connection profile exists, but is the wrong type	The Connection Profile Name provided for <property> does not match the selected type for <type property>. Please provide a valid Connection Profile.

Staging path scheme

ExternalStagingLocation is checked against S3 Tables Writer's data lake type, which is always S3 Tables, so the path must use the s3a:// scheme. A schemeless path (for example my-bucket/stage) is silently corrected by prepending s3a:// rather than rejected.

Condition	Compiler error
ExternalStagingLocation uses a different, unsupported scheme (for example gs:// or hdfs://)	Unsupported file scheme for the property 'ExternalStagingLocation'. Expected path starting with s3a://

Write mode compatibility

Condition	Compiler error
ParallelThreads > 0 while Mode is MERGE	Specified configuration is not valid. Parallel Threads are allowed only in the 'APPENDONLY' mode.
OptimizedMerge is true while Mode is APPENDONLY	Specified configuration is not valid. OptimizedMerge is supported only in the 'MERGE' mode.

Catalog type compatibility

CatalogConnectionProfileName must be empty when CatalogType is S3Tables, and must be set when CatalogType is AWSGlueCatalog.

Condition	Compiler error
CatalogType=S3Tables and CatalogConnectionProfileName is non-blank	Catalog Connection Profile should not be provided for Hadoop Catalog.
CatalogType=AWSGlueCatalog and CatalogConnectionProfileName is blank	The Catalog Connection Profile Name should be provided for AWSGlueCatalog Catalog.