PostgreSQL initial load

Use Database Reader for initial load to create a point-in-time copy of your existing source PostgreSQL dataset at the target. This establishes a baseline for subsequent continuous updates. You can use Database Reader with PostgreSQL 9.4.x and later versions and with Amazon RDS for PostgreSQL, Azure Database for PostgreSQL, Azure Database for PostgreSQL - Flexible Server, Crunchy Postgres, Google AlloyDB for PostgreSQL, Google Cloud SQL for PostgreSQL, and Neon.

You can leverage Database Reader's parallel processing capabilities to accelerate data loading for large datasets, while the system automatically handles schema creation and data type mapping between PostgreSQL and target systems. You can configure table selection with wildcard support, create custom SQL queries for selective data extraction, optimize batch sizes for performance tuning, and establish SSL connectivity for secure connections.

The required PostgreSQL JDBC driver is bundled with Striim Platform.

If not using an automatic pipeline wizard, before running the initial load application, see Switching from initial load to continuous replication of PostgreSQL sources.

PostgreSQL setup for initial load

In PostgreSQL, create a user for use by Striim with SELECT privileges on the tables to be read.

Database Reader properties for PostgreSQL sources

property	type	default value	notes
Connection Profile Name	enum		Not applicable to PostgreSQL in this release.
Connection URL	String		Specify `jdbc:postgresql://<ip address>:<port>/<database name>?reWriteBatchedInserts=true`.
Create Schema	Boolean	False	If set to True, when Database Reader's output stream is the input stream of an Azure Synapse Writer, BigQuery Writer, Database Writer (for MariaDB, MariaDB Xpand, MySQL, Oracle, PostgreSQL, SQL Server, or YugabyteDB only), Databricks Writer, Fabric Data Warehouse Writer, Fabric Mirror Writer, Iceberg Writer, Snowflake Writer, or Spanner Writer target, the schema(s) and tables specified in the Tables property in the writer will be created in the target. For example, if the Tables property in the writer is `sourceschema1.%,targetschema1.%;sourceschema2.%,targetschema2.%` then the schemas targetschema1 and targetschema2 will be created in the target, and all the tables in sourceschema1 and sourceschema2 will be created in the corresponding target schema. Data types will be mapped as per Target data type support & mapping for PostgreSQL sources. Single-column PRIMARY KEY constraints will be replicated in the target. DEFAULT, FOREIGN KEY, NOT NULL, composite PRIMARY KEY, and UNIQUE KEY constraints will not be replicated. If a table already exists in the target, Striim will verify that the source and target structure match. If they do not, the application will halt and log a TargetTableSchemaMismatchException. MON output for the target tables will include a schemaCreationStatus of Pending, InProgress, Success, or Failed.
Database Provider Type	String	Default	Set to Postgres. Controls which icon appears in the Flow Designer.
Enable Partition-Aware Read	Boolean	Trie	See Fast Snapshot Loading.
Excluded Tables	String		Data for any tables specified here will not be returned. For example, if `Tables` uses a wildcard, data from any tables specified here will be omitted. Multiple table names (separated by semicolons) and wildcards may be used exactly as for `Tables`.
Fetch Size	Integer	100	Sets the maximum number of records to be fetched from the database in a single JDBC method execution (see the discussion of fetchsize in the documentation for your JDBC driver).
JAAS Configuration	String		If you Enable Kerberos authentication, use this property to configure it. To authenticate, , specify `java.security.krb5.realm=<Kerberos realm>; java.security.krb5.kdc=<Kerberos server IP address or network name>; javax.security.auth.useSubjectCredsOnly=false; java.security.auth.login.config=/<path>/login.conf`. For example, `java.security.krb5.realm=MYDOMAIN.COM; java.security.krb5.kdc=kdc.mydomain.com; javax.security.auth.useSubjectCredsOnly=false; java.security.auth.login.config=/home/Jdbc/Security/login.conf`. You must also append the appropriate application name from the specified '`login.conf` (see Enable Kerberos authentication) to the connection URL. For example, `jdbc:postgresql://<ip address>:<port>/<database name>&jaasApplicationName=<application name>`.
MultiQuery	String		See Fast Snapshot Loading.
Password	encrypted password		The password for the specified user. See Encrypted passwords.
Parallel Threads	Integer	1	See Fast Snapshot Loading.
Query	String		If using the Flow Designer, see Table selection in Flow Designer. The following is relevant when using TQL. Optionally, specify a single SQL SELECT statement specifying the data to return. You may query tables, aliases, synonyms, and views. Query is not supported when Create Schema is True, Parallel Threads is greater than 1, or Restart Behavior on IL Interruption is Truncate_target_table or Replace_target_table. When `Query` is specified and `Tables` is not, the WAEvent TableName metadata field value will be QUERY. When both `Query` and `Tables` are specified, the data specified by `Query` will be returned, and the `Tables` setting will be used only to populate the TableName field. Wildcards in table names, such as `SELECT ... FROM hr.`, are not supported. Specify multiple table names separated by commas. If the query includes a synonym containing a period, it must be enclosed in escaped quotes. For example: `select from \"synonym.name\"` If using a query when the output of a DatabaseReader source is the input of a DatabaseWriter target, specify the target table name as the value of DatabaseReader's Tables field.
Quiesce on IL Completion	Boolean	False	With the default value of False, you must stop the application manually after all data has been read. Set to True to automatically quiesce the application after all data has been read (see discussion of QUIESCE in Console commands). When you see on the Apps page that the application is in the Quiescing state, it means that all the data that existed when the query was submitted has been read and that the target(s) are writing it. When you see that the application is in the Quiesced state, you know that all the data has been written to the target(s). At that point, you can undeploy the initial load application and then start another application for continuous replication of new data.Console commands Note Set to True only if all targets in the application support auto-quiesce (see Writers overview).Writers overview
Restart Behavior on IL Interruption	Enum	Keep target table data	See Fast Snapshot Recovery during initial load.
Return DateTime As	String	Joda	Set to `joda` to return timestamp values as Joda timestamps. Set to `String` to return timestamp values as strings. The primary purpose of this option is to avoid losing precision when microsecond timestamps are converted to Joda milliseconds. The format of the string is yyyy-mm-dd hh:mm:ss.ffffff.
SSL Config	String		Not applicable to PostgreSQL in this release.
Tables	String		If using the Flow Designer, see Table selection in Flow Designer for discussion of Visual Mode, List Mode, and Query Mode. The following is relevant when writing TQL. Specify the table(s) or view(s) to be read. Specify names as `<schema name>.<table name>`. Names are case-sensitive. You may specify multiple tables and views as a list separated by semicolons or with the `%` wildcard. For example, `HR%` would read all the tables whose names start with HR. You may use the `%` wildcard only for tables, not for schemas or databases. The wildcard is allowed only at the end of the string: for example, `mydb.prefix%` is valid, but `mydb.%suffix` is not. If you are using the Query property, specify QUERY as the table name. Modifying this property can interfere with recovery. If recovery is enabled for the application and it is necessary to modify the tables list, export the application (see Apps page), drop it, update the exported TQL with the new values, and import it (see Apps page).
Use Connection Profile	Boolean	False	Not applicable to PostgreSQL in this release.
Username	String		Specify the DBMS user name the adapter will use to log in to the server specified in ConnectionURL. This user must have SELECT privileges on the tables specified in the Tables property.
Vendor Configuration	Striing		Not applicable to PostgreSQL in this release.

Sample Database Reader WAEvent

For the following row:

id  first_name  last_name  phone  street          city          state  zip_code
1   Deborah     Burks      NULL   9273 Thorne AV  Orchard Park  NY     14127

The WAEvent would be similar to:

WAEvent{
  data: [1,"Deborah","Burks",null,"9273 Thorne AV","Orchard Park","NY","14127"]
  metadata: {"TableName":"BikeStores.sales.customers","ColumnCount":8,
    "OperationName":"SELECT","OPERATION_TS":1681412863364}
  userdata: null
  before: null
  dataPresenceBitMap: "fwM="
  beforePresenceBitMap: "AAA="
  typeUUID: {"uuidstring":"01edda2e-77f7-9b21-83c2-8e859085da65"}
};

The operation name for Database Reader WAEvents is always SELECT.

For more information about WAEvent, see WAEvent contents for change data.

Database Reader data type support and correspondence for PostgreSQL sources

Database Reader uses PostgreSQL's JDBC driver to read from PostgreSQL. The source types are those returned by the JDBC driver. For more information, see PostgreSQL's JDBC documentation.

JDBC column type	TQL type
Types.ARRAY	java.lang.String
Types.BIGINT	java.lang.Long
Types.BIT	java.lang.Boolean
Types.CHAR	java.lang.String
Types.DATE	org.joda.time.LocalDate
Types.DECIMAL	java.lang.String
Types.DOUBLE	java.lang.Double
Types.FLOAT	java.lang.Double
Types.INTEGER	java.lang.Integer
Types.NUMERIC	java.lang.String
Types.REAL	java.lang.Float
Types.SMALLINT	java.lang.Short
Types.TIMESTAMP	org.joda.time.DateTime
Types.TINYINT	java.lang.Short
Types.VARCHARCHAR	java.lang.String
other types	java.lang.String

PostgreSQL initial load

PostgreSQL setup for initial load

Database Reader properties for PostgreSQL sources

Note

Sample Database Reader WAEvent

Database Reader data type support and correspondence for PostgreSQL sources

Search results