Skip to main content

Switching from initial load to continuous replication of Salesforce sources

Using Salesforce automated pipeline wizards

Automated pipeline wizards automatically create two applications, one for initial load and another for continuous replication, and switch between them automatically. See Introducing automated pipelines for more information.

  • To perform initial load using Salesforce Reader, then automatically switch to continuous real-time replication using Salesforce CDC Reader, use a wizard with a Salesforce CDC source.

  • To perform initial load using Salesforce Reader, then automatically switch to continuous incremental replication using Salesforce Reader, use a wizard with a Salesforce source.

Using Salesforce Reader's Automated mode

In Automated mode, Salesforce Reader uses a ‘startTimestamp’ of ‘1970-01-01’. In this mode, it first performs initial load, then switches to continuous incremental replication, capturing new records as scheduled by the Polling Interval. This requires only a single application.

Switching from initial load to continuous real-time replication of Salesforce sources manually

Before running the initial load application, run the following command on the source database using an appropriate database client and record the value returned:

When creating the continuous real-time replication application:

  1. Enable recovery.

  2. In Salesforce CDC Reader, set Start Position to the ReplayID you recorded before performing the initial load .

  3. If the target writer has an Ignorable Exception Code property, set it as follows:

    DUPLICATE_ROW_EXISTS, NO_OP_UPDATE, NO_OP_DELETE

  4. If the target writer has an Ignorable Exception Code property, when you know all open transactions have completed and been written to the target, stop and undeploy the application, edit the target, clear the Ignorable Exception Code value, save, and deploy and start the application. Since recovery is enabled, writing will resume where it left off, and there should be no missing or duplicate rows.

Switching from initial load to continuous incremental replication of Salesforce sources manually

Do the following to make continuous incremental replication pick up where initial load left off:

  1. After initial load completes, query the target to get the timestamp of the most recent event written.

  2. In Salesforce Reader, set the Start Timestamp value to one second after that timestamp.