Skip to main content

Building pipelines with ServiceNow Reader

You can read data from ServiceNow using the ServiceNow Reader and write to any target supported by Striim.

We recommend that you create a single Striim application using the Reader in Automated mode. In this mode, the Reader performs an initial load of existing data and automatically transitions to continuous incremental replication after the initial load has completed. For foundational concepts about pipeline phases, see Pipelines.

Reader modes

The ServiceNow Reader supports three modes.

Mode

Description

When to use

Wizard

Flow Designer, TQL, API

Automated (recommended)

Performs an initial load of the selected tables and then automatically transitions to continuous incremental replication without requiring manual intervention.

Use this mode for most pipelines, especially when you are building a new pipeline.

Yes

Yes

Initial Load

Performs a one-time snapshot of the selected tables and does not continue with continuous incremental replication after completion.

Use this mode when you need to run the initial load separately from incremental replication or to perform an ad hoc resynchronization of the source data.

No

Yes

Incremental Load

Continuously replicates new or updated records using a configured watermark field and polling interval.

Use this mode if the initial load has already been completed and you want to replicate only new or updated data going forward.

No

Yes

Two ways to build a ServiceNow pipeline

Before creating a pipeline, complete the required setup and configuration steps for ServiceNow as described in Initial setup.

You can build pipelines in one of two ways: using the pipeline wizard (recommended), or manually using Flow Designer, TQL, or the REST API.

Using the pipeline wizard (recommended)

If you are building near-real-time pipelines from ServiceNow to supported targets, we recommend using a wizard from the Create App page to build an application that reads from ServiceNow. The wizard automatically:

  1. Creates a single application using the Reader in Automated mode.

  2. Configures both initial load and continuous incremental replication.

  3. Creates the target schema and tables that match the selected tables.

  4. Copies existing data during the initial load phase.

  5. Automatically transitions to incremental replication when the initial load completes.

  6. Configures watermark tracking and polling intervals.

The wizard simplifies setup, promotes best practices, and reduces configuration errors.

Creating a pipeline manually

If your use case or organizational policies do not allow the use of the pipeline wizard (for example, you are using a CI/CD system to move configurations to production) or if the wizards do not support your target, you can manually create and configure an application using Flow Designer, TQL, or the Striim REST API.

When creating a pipeline manually, set the Reader's Mode property to Automated. This allows a single application to handle both initial load and continuous replication without requiring changes between phases. You may also use the Reader in Initial Load or Incremental Load mode when your architecture requires separate applications for each phase.

Configuration guidelines

When configuring the ServiceNow Reader, follow these best practices.

  1. Set the Reader mode. Choose the mode that aligns with your use case. We recommend that you start with Automated mode.

  2. Set the Incremental Load Marker. The Incremental Load Marker is the field the reader uses as the watermark for incremental replication. It must be a datetime field whose value increases with every change to the record and is never empty. A field that does not increase strictly, or that contains empty values, can cause records to be missed.

    The default Incremental Load Marker for all ServiceNow tables is sys_updated_on. To use a different field, specify one or more table-and-field pairs in the format <table_name>=<field_name>, separating pairs with semicolons. Use the % wildcard to set a default for all tables and override it for individual tables. For example, the following sets sys_updated_on as the marker for every table except change_request, which uses opened_at:

    %=sys_updated_on;change_request=opened_at

    If you specify a marker field that a table does not contain, the application throws an exception.

    Note

    Insert and update operations are distinguished only when the marker is sys_updated_on, which the reader compares against sys_created_on to determine whether a record is new. When you specify any other datetime field as the marker, this distinction is not available and every record is emitted as an insert.

  3. Configure schema creation. If the required schemas do not already exist on the target and you want Striim to create them automatically, enable Migrate Schema. If any tables from the source schema already exist on the target, this operation will fail. If you prefer not to have Striim create target schemas, pre-create target tables using native or third-party utilities and ensure that the target schema is compatible with the source schema. Leave Migrate Schema disabled in this case.

  4. Configure continuous replication. Set the Polling Interval and configure the target for upsert semantics using appropriate key columns.

  5. Set Start Position (optional). If you need to resume replication from a specific point, provide the last successful watermark value as the Start Position.

Create a ServiceNow Reader application using the Flow Designer

This procedure outlines how to use Striim's Flow Designer to build and configure data pipelines. Flow Designer enables you to visually create applications with minimal or no coding.

  1. Go to the Apps page and click Start from scratch.

  2. Provide the Name and Namespace for your app.

  3. In the component section, expand Sources, search for ServiceNow Reader, and select it.

  4. In the properties panel, enter connection and runtime properties (for example, Connection URL, authentication, Tables, Mode, Incremental Load Marker, Polling interval). Set Mode to Automated to perform initial load followed by continuous incremental replication.

  5. Click Save. Drag processors/enrichers and a target to complete the pipeline. Deploy and start the application.

Create a ServiceNow Reader application using TQL

The following sample TQL uses the ServiceNow Reader to read from ServiceNow tables using Initial Load mode. For most pipelines, set Mode to Automated to perform initial load and then automatically transition to incremental replication.

CREATE APPLICATION ServiceNow_App;

CREATE SOURCE sn_source USING Global.ServiceNowReader (
  Mode: 'InitialLoad',
  PollingInterval: '120s',
  IncrementalLoadMarker: '%=sys_updated_on;change_request=opened_at',
  ClientSecret: '<client-secret>',
  Password: '<password>',
  Tables: 'incident;problem',
  Username: '<username>',
  ClientId: '<client-id>',
  ConnectionUrl: 'https://<instance>.service-now.com/',
  FetchSize: 10000,
  MaxConnections: 20,
  ThreadPoolCount: 10,
  MigrateSchema: true
)
OUTPUT TO sn_stream;

END APPLICATION ServiceNow_App;