S3 Tables Writer operational considerations
Retry and recovery behavior
S3 Tables Writer offers connection retry configurable at each component's own connection profile (AWS EMR, S3, and S3Tables — see Connection retry support in S3 Tables Writer programmer's reference).
S3 Tables Writer provides recovery capabilities similar to other Striim data warehouse writers, such as the Snowflake and BigQuery writers. During a restart, events from previous runs may be reprocessed and written to the Iceberg tables, maintaining the same order in which they were originally processed. S3 Tables Writer provides at-least-once processing semantics — after a restart, expect some previously written events to be reprocessed rather than silently skipped.
Initial load recovery: supported at table-level granularity. A table that had already completed loading before a restart is not rewritten.
CDC / merge recovery: at-least-once semantics apply; events are reprocessed in original order after a restart.
Optimized merge: when enabled, reduces the data volume S3 Tables Writer must reprocess for partial-image update and delete operations during normal CDC processing (see Optimized merge in S3 Tables Writer programmer's reference).
Best practices
Performance
Use higher event counts (for example 50000 or more) for high-volume CDC; use smaller event counts and shorter intervals when you need lower latency.
Size the EMR cluster for your expected volume and throughput; consider spot instances with an on-demand fallback for cost efficiency.
Use parallel threads for initial snapshot loads; size the thread count to your table count and cluster capacity, and monitor cluster utilization.
Enable Optimized Merge for CDC loads when the source supports partial-image updates.
Group related tables in the same application; avoid configuring an upload policy that produces excessive numbers of small batches.
Security
Prefer IAM roles over explicit access keys; if you do use access keys, rotate them regularly and apply least-privilege permissions.
Rely on S3 Tables' built-in encryption at rest; ensure TLS is used in transit; consider AWS KMS for additional key management where required.
Use VPC endpoints for S3 and S3 Tables where possible; restrict EMR security groups; use private subnets where appropriate.
Operational
Monitor adapter metrics; alert on connection failures and batch errors; watch EMR utilization and cost.
Test recovery scenarios before production use; ensure sufficient checkpoint storage; document your recovery procedures.
Test with representative data volumes; validate DDL propagation; test failover and recovery before going live.
Review upload policies periodically; monitor S3 Tables storage costs and lifecycle policies; keep the EMR cluster's Iceberg libraries current.
Choose the S3 Tables embedded catalog for simplicity and performance; choose AWS Glue Catalog when other AWS services or tools need to query the table metadata through Glue.
Data quality
Use Initial Schema Creation for new tables; validate data types before migration; test DDL propagation in a non-production environment first.
Specify key columns explicitly for sources that lack a usable primary key; validate uniqueness before starting CDC loads; use composite keys where needed.
Use column mapping to handle schema differences between source and target; document your mappings; test with sample data first.
S3 Tables Writer monitoring metrics
S3 Tables Writer exposes two categories of monitoring metrics: table-level Table Write Info, and adapter-level metrics that summarize activity across all tables the application writes to.
Table Write Info
A JSON array of monitoring metrics calculated and displayed at table-level granularity.
Sub-metric | Description | Frequency |
|---|---|---|
Mapped Source Table | The source table the target table is mapped to. | Per batch |
Last batch info | Granular metrics from the last executed batch, including Batch Sequence Number, Batch Event Count, Batch Size (bytes), Max Record Size, Batch Accumulation Time, Micro Batch Count, and Integration Task Time. | Per batch |
Last successful merge time | Time of the last executed task for the table. | Per batch |
Last Applied DDL Time | Time of the last executed DDL for the table. | Per DDL batch |
Last Applied DDL Statement | The last executed DDL statement for the table. | Per DDL batch |
Total Batches Created / Queued / Ignored / Uploaded | Batch counters for the table. | Per batch |
Total event info | Overall event count for the table. | Per batch |
Avg Upload / Compaction / In-Mem Compaction / Merge / Waiting-in-Queue Time (ms) | Average timings across all batches. | Per batch |
Avg Event Count Per Batch, Avg Batch Size (bytes) | Average batch shape. | Per batch |
Avg Stage Resources Management Time (ms) | Average time to clear or create staging resources. | Per batch |
Avg / Min / Max Integration Time (ms) | Time for a processed batch to reach the target table. | Per batch |
Note: Last Applied DDL Statement is restored after a restart only for a single statement. This field can be left empty after a restart when multiple DDL statements were applied in a single batch before the restart. Treat this as a known restore-on-restart limitation for that specific field, not a general failure of DDL tracking.
Adapter level metrics
Metric | Description | Frequency |
|---|---|---|
Write Timestamp | Last time a batch was executed across all tables. | Per batch |
Target Freshness | Time since a batch was executed across all tables. | Per batch |
Discarded Event Count | Total events discarded across all tables (for example, events for excluded tables). | Per event / batch |
Connection Retry Information | Total reconnects and the last known reconnect time. | — |
Queued Batches Size In Bytes | Size of all queued batches. | — |
Limitations
S3 Tables naming constraints. Table bucket names, table names, and column names must comply with AWS S3 Tables naming restrictions: only lowercase letters, numbers, and underscore are supported. Special characters, spaces, and uppercase letters are not supported and can cause DDL operations such as CREATE TABLE to fail.
CREATE TABLE with the AWS Glue catalog requires an AWS-supplied patch. Without the patch, CREATE TABLE against the AWS Glue catalog fails on EMR because EMR's Hadoop configuration ships with pre-configured Glue settings that override the writer's REST catalog configuration, causing Iceberg to use AWS's GlueCatalog implementation instead — which expects a traditional S3 warehouse path rather than the account-id:s3tablescatalog/bucket format S3 Tables uses. AWS has provided a fix consisting of a patched Iceberg RPM and a bootstrap script applied when you create the EMR cluster; this is a third-party AWS artifact that Striim does not bundle or publicly host. Contact Striim Support for guidance if you need CREATE TABLE DDL propagation on the Glue catalog path.
Wildcard table mappings can break against S3 Tables' lowercase-only naming. Many RDBMS sources, including Oracle by default, use uppercase identifiers, so a wildcard table mapping such as Tables:'SCOTT.%' will attempt to create uppercase-named target tables and fail. Use explicit, static, lowercase table mappings instead of wildcards when the source uses uppercase identifiers.
AWS enforces region-level API rate limits on S3 Tables. Striim automatically retries throttled S3 Tables API calls with an internal exponential back-off policy, but under high traffic or with many concurrent S3 Tables Writer applications in the same account and region, retries can still be exhausted, adding latency. This is an AWS platform constraint, not a Striim defect. Monitor for throttling and request an AWS API quota increase proportional to your S3 Tables application count and data volume.