Skip to main content

Partition-aware parallelism

Partition-aware parallelism allows Database Reader to detect a partitioned table in the source and read the detected partitions individually or concurrently, if Parallel Threads is set to 2 or more.

Prerequisites

  • The database user must be able to read the selected tables and access the database metadata used to discover their partitions.

Configure partition-aware reads in the Flow Designer

  • In the Database Reader properties, under Fast Snapshot Loading, enable Partition-Aware Read to True.

How partition-aware reads work

  • Partition-aware reads apply only to partitioned source tables. If a table is not partitioned, Database Reader can still read the table, but partition-aware reads do not provide source-read acceleration.

  • When Parallel Threads is set to 1, Database Reader reads table partitions sequentially.

  • When Parallel Threads is set to 2 or more, Database Reader can use available threads to read different table partitions concurrently.

  • Database Reader emits rows from the original source table. The table partitioning is used only as an internal read strategy.

Note: Partition-aware reads operate at the table level and read every partition. To read only a specific partition or a subset of rows, use Query or Multi Query with database-specific SQL.

TQL example: Enable partition-aware reads

CREATE SOURCE OrdersSnapshot USING Global.DatabaseReader (
  ConnectionURL: 'jdbc:postgresql://<host>:5432/<database>',
  DatabaseProviderType: 'PostgreSQL',
  Username: '<username>',
  Password: '<password>',
  Tables: 'public.orders',
  FetchSize: 1000,
  ParallelThreads: 4,
  EnablePartitionAwareRead: true,
  QuiesceOnILCompletion: true
)
OUTPUT TO OrdersSnapshotStream;

WAEvent metadata

Database Reader preserves the original table identity in the event metadata and adds the partition name. You can use PartitionName in continuous queries, routers, or monitoring logic when you need partition-level context.

metadata: {
  "PartitionName": "public.orders_2026_01",
  "TableName": "public.orders",
  "EntityName": "public.orders",
  "OperationName": "SELECT"
}

Partition-aware recovery

Recovery remains table-based even though Database Reader reads a partitioned table one partition at a time. Striim marks the table in progress when any partition starts and marks it completed only after every partition finishes.

  • After a restart, Database Reader skips tables that completed before the interruption.

  • If a partitioned table was incomplete, Database Reader rereads every partition of that table.

  • Restart Behavior on IL Interruption applies to the table, not to individual partitions.

Observability: Mon metrics

The Table Info metric includes partition-level status and rows-read information. Parallel-thread metrics identify the table or partition currently assigned to a thread and retain a history of completed work.

  • Completed: Database Reader finished reading the partition.

  • In progress: A thread is currently reading the partition.

  • Pending: The partition is waiting for an available thread.

Operational considerations

  • Partition-level recovery is not supported. Select a restart behavior that prevents partially loaded target data from causing duplicates or conflicts when an incomplete table is read again