Skip to main content

Using Preserve Source Transaction Boundary with Spanner targets

When Preserve Source Transaction Boundary has its default value of False, transactions in the Spanner target (see Spanner > Documentation > Guides > Transactions overview) may contain operations from multiple transactions in the source database. Set Preserve Source Transaction Boundary to True to ensure that each transaction in the Spanner target will contain operations from only a single source database transaction or two or more complete source database transactions.

Note

To use Preserve Source Transaction Boundary, Filter Transaction Boundaries must be set to False in the source.

Preserve Source Transaction Boundary may be used when the source is:

  • GG Trail Reader

  • MariaDB Reader

  • MariaDB Xpand Reader

  • MSJet

  • MS SQL Reader with Transaction Support set to True

  • MySQL Reader

  • OJet

  • Oracle Reader

  • PostgreSQL Reader

When Preserve Source Transaction Boundary is True, a single source transaction will result in multiple transactions in the Spanner target in the following circumstances.

  • When the Batch Policy Event Count or Interval is reached, one or more complete source transactions are committed in the target as a single transaction, and a new target transaction is begun.

  • When a DDL event is received from the source, any current target transaction is committed, the DDL is committed as a separate transaction, and a new transaction is begun for any remaining operations from the source transaction.

  • When the target transaction reaches Spanner's "Mutations per commit" limit, the target transaction is committed and a new target transaction is begun. See Spanner > Documentation > Resources> Quotas & limits > Limits for creating, reading, updating, and deleting data for the current limit (as of August 2024 it was 80,000) and "How can I estimate mutation counts?" in Cloud Spanner doubles the number of updates per transaction (note that this older blog post refers to a previous limit of 40,000). Google may allow you to increase this limit; if you do, Contact Striim support to have your Striim environment adjusted accordingly.

Batch Policy interacts with Preserve Source Transaction Boundary as follows:

TQL

results

Batch Policy is disabled. (Not supported if the source's Filter Source Transaction property (if any) is True.)

Sample TQL: BatchPolicy: ‘-1', PreserveSourceTransactionBoundary: 'true'

Target transactions match source transactions (unless a DDL event is encountered).

Batch Policy has Event Count and Interval, and no single transaction exceeds the Event Count or Interval.

Sample TQL: BatchPolicy: ‘eventCount: 1000, Interval: 60s', PreserveSourceTransactionBoundary: 'true'

Target transactions contain one or more complete source transactions (unless mutations per commit limit is exceeded or a DDL event is encountered).

Batch Policy has Event Count and Interval, and a single source transaction exceeds one or both.

Sample TQL: BatchPolicy: ‘eventCount: 1000, Interval: 60s', PreserveSourceTransactionBoundary: 'true'

Batch policy is ignored; target transactions match source transactions (unless mutations per commit limit is exceeded or a DDL event is encountered).

Batch Policy has only Event Count.

Sample TQL: BatchPolicy: ‘eventCount: 1000', PreserveSourceTransactionBoundary: 'true'

Not supported.

Batch Policy has only Interval.

Sample TQL: BatchPolicy: ‘Interval: 60s', PreserveSourceTransactionBoundary: 'true'

Batching is performed based on the specified interval.