Fast Snapshot Loading
Fast Snapshot Loading accelerates Database Reader snapshot loads by increasing the amount of source data that can be read in parallel. The feature includes three source-read capabilities: reading multiple tables concurrently, running multiple labeled queries, and reading table partitions in parallel. Depending on the source database and loading scenario, you may use one, two, or all three capabilities to increase source-read throughput.
Use Fast Snapshot Loading for initial loads, backfills, and historical bootstraps. Moving historical data faster helps downstream data warehouses, data lakes, databases, and AI context or feature stores begin operating with a complete baseline sooner.
Additionally, for all sources, in the event of initial load being interrupted, Striim does not re-read tables it already read (Fast Snapshot Recovery).
Note
While Fast Snapshot Loading can speed up the source-read phase, the end-to-end load time depends on additional factors including source database system resources, table size, network throughput, data pipeline architecture, target writer configuration, and the target data system resources.
Fast Snapshot Loading capabilities
Fast Snapshot Loading is an umbrella feature that includes three source-read capabilities: Database Reader read parallelism, multiple labeled queries, and partition-aware parallelism.
Capability | How it works | Supported sources |
Read parallelism | Read multiple tables concurrently. Each table is read by one thread. | All sources supported by Database Reader |
Multiple labeled queries | Runs up to 20 SELECT queries sequentially to read from the source database, or concurrently, if read parallelism is enabled. | MySQL, Oracle, PostgreSQL, and SQL Server |
Partition-aware parallelism | Detects source table partitions and reads the partitions sequentially or concurrently, if read parallelism is enabled. | MySQL, Oracle, PostgreSQL, and SQL Server |