Database Reader
You can use Database Reader to read all the data that exists in a supported database or data warehouse at the time the Striim application containing Database Reader is started. The most common use case for Database Reader is to load this existing data to the target to establish an initial, point-in-time copy of the source data that can serve as a starting point for subsequent continuous updates or syncs. This process is called "initial load" in Striim, and "historical sync" or "initial snapshot" by others. After completing the initial load using Database Reader, you can use a CDC reader (if available) or Incremental Batch Reader to continuously load updates to the source database or data warehouse to the target.
Database reader supports the following sources:
BigQuery (see Runtime considerations when using Database Reader or Incremental Batch Reader with BigQuery sources)
DB2 (see Runtime considerations when using Database Reader with Db2 sources)
HP NonStop SQL/MX (and SQL/MP via aliases in SQL/MX) versions 3.2.1 to 3.8.1
MariaDB (including Amazon RDS for MariaDB, MariaDB Galera Cluster, MariaDB Xpand, and SkySQL) versions 10.1 and later
MySQL (including Amazon Aurora for MySQL, Amazon RDS for MySQL, Azure Database for MySQL, and Google Cloud SQL for MySQL) versions 5 and 8
Oracle (including Amazon RDS for Oracle) versions 11G, 12C, 18C, 19C, and 21C
PostgreSQL (including Amazon Aurora with PostgreSQL compatibility, Amazon RDS for PostgreSQL, Azure Database for PostgreSQL, Azure Database for PostgreSQL - Flexible Server, Crunchy Postgres, Google AlloyDB for PostgreSQL, Google Cloud SQL for PostgreSQL), and Neon, versions 8.4 to 17
Snowflake
SQL Server (including Amazon RDS for SQL Server and Azure SQL Database) versions 2012, 2014, 2016, and 2017
Sybase ASE versions 16.0 and later (Linux)
Teradata
YugabyteDB versions 2.18 and later
Note
Most sources supported by Database Reader are also supported by Incremental Batch Reader.
If the connection to the database is interrupted, the application will halt.
For all databases, when this adapter is deployed to a Forwarding Agent, the appropriate JDBC driver must be installed as described in Installing third-party drivers in the Forwarding Agent.
If you are using Database Reader to perform an initial load before running an application using a CDC reader, enable CDC on the source and perform the setup tasks for the CDC reader before starting the initial load.
Performing initial loads with Database Reader
Fast Snapshot Recovery during initial load
When a Database Reader application with recovery enabled is stopped, halts, or terminates before the initial load has completed, when you restart the application it will resume reading tables that have not been completely written to the target. Database Reader's Restart Behavior on IL Interruption setting gives you three choices for what the writer will do:
Keep target table data (default): The application will resume loading, from the beginning, the source tables that had not yet been fully written to the target before the application was interrupted, appending the data to any target tables with partial data. This will typically result in duplicate rows in the target or the application halting with a duplicate row error, which will require manual effort (described below) to resolve. This is the default because it is the closest to the behavior of Striim 4.x, in which recovery was not supported for Database Reader.
Replace target table (recommended): Striim will drop and re-create any target tables with partial data or no data, then resume loading, from the beginning, the source tables that had not yet been fully written to the target before the application was interrupted. This option requires Create Schema to be True, and the writer must have DROP permission on tables in the target schema.
Truncate target table: Striim will delete all data from target tables with partial data or no data, then resume loading, from the beginning, the source tables that had not yet been fully written to the target before the application was interrupted. The writer must have TRUNCATE permission on tables in the target schema.
Functionally, Truncate target table is similar to Replace target table except it does not require you to give Striim permission to drop tables in the target.
If you create the Database Reader application using a wizard, to change the Restart Behavior on IL Interruption setting you must modify the application in the Flow Designer.
In detail, depending on the initial load status of each source table, after the application is restarted Striim will do the following or, in the case of Keep target table data, you must do the following.
table status: description | Keep target table data | Replace target table | Truncate target table |
---|---|---|---|
WAITING_TO_PROCESS: Database Reader had not read the table before the application was interrupted. | If Create Schema is True, you must manually drop the table before restarting the application. If Create Schema is False, you must manually truncate the target table before restarting the application. | Striim will drop and re-create the target table. | Striim will attempt to truncate the target table. If the table does not exist, the application will halt. |
SCHEMACREATION_COMPLETED: Create Schema is True and the table has been created in the target, but Database Reader had not started reading the table's data before the application was interrupted. | You must manually truncate the target table before restarting the application. (Writing data may have begun even though the status had not yet changed to IL_INPROGRESS.) | Striim will drop and re-create the target table | Striim will truncate the empty target table. |
IL_INPROGRESS: Database Reader was reading the table when the application was interrupted. | You must manually truncate the target table before restarting the application. | Striim will drop and re-create the target table | Striim will attempt to truncate the target table. If the table does not exist, the application will halt. |
IL_COMPLETED: The table was fully written to the target before the application was interrupted. | No action required. | Database Reader will not read the table again. | Database Reader will not read the table again. |
Creating a Database Reader application
Creating a Database Reader application using TQL
The following example creates a cache of data retrieved from a MySQL table:
CREATE TYPE RackType( rack_id String KEY, datacenter_id String, rack_aisle java.lang.Integer, rack_row java.lang.Integer, slot_count java.lang.Integer ); CREATE CACHE ConfiguredRacks USING DatabaseReader ( ConnectionURL:'jdbc:mysql://10.1.10.149/datacenter', Username:'username', Password:'passwd', Query: "SELECT rack_id,datacenter_id,rack_aisle,rack_row,slot_count FROM RackList" ) QUERY (keytomap:'rack_id') OF RackType; The following example creates a cache of data retrieved from an Oracle table: CREATE TYPE CustomerType ( IPAddress String KEY, RouterId String, ConnectionMode String, CustomerId String, CustomerName String ); CREATE CACHE Customers USING DatabaseReader ( Password: 'password', Username: 'striim', ConnectionURL: 'jdbc:oracle:thin:@node05.example.com:1521:test5', Query: 'SELECT ip_address, router_id, connection_mode, customer_id, customer_name FROM customers', FetchSize: 1000 ) QUERY (keytomap:'IPAddress') OF CustomerType;
Database Reader programmer's reference
Database Reader properties
property | type | default value | notes |
---|---|---|---|
Connection Profile Name | enum | Appears in Flow Designer only when Database Provider Type is BigQuery and Use Connection Profile is True. See Introducing connection profiles. | |
Connection URL | String | When connecting through an SSH tunnel (see Using an SSH tunnel to connect to a source or target), specify the IP address of the tunnel.
| |
Create Schema | Boolean | False | If set to True, when Database Reader's output stream is the input stream of an Azure Synapse Writer, BigQuery Writer, Database Writer (for MariaDB, MariaDB Xpand, MySQL, Oracle, PostgreSQL, SQL Server, or YugabyteDB only), Databricks Writer, Fabric Data Warehouse Writer, Fabric Mirror Writer, Iceberg Writer, Snowflake Writer, or Spanner Writer target, the schema(s) and tables specified in the Tables property in the writer will be created in the target. For example, if the Tables property in the writer is Data types will be mapped as per Data type support & mapping for schema conversion & evolution. Single-column PRIMARY KEY constraints will be replicated in the target. DEFAULT, FOREIGN KEY, NOT NULL, composite PRIMARY KEY, and UNIQUE KEY constraints will not be replicated. If a table already exists in the target, Striim will verify that the source and target structure match. If they do not, the application will halt and log a TargetTableSchemaMismatchException. MON output for the target tables will include a schemaCreationStatus of Pending, InProgress, Success, or Failed. |
Database Provider Type | String | Default | Controls which icon appears in the Flow Designer and whether Use Connection Profile appears in the web UI. |
Excluded Tables | String | Data for any tables specified here will not be returned. For example, if | |
Fetch Size | Integer | 100 | Sets the maximum number of records to be fetched from the database in a single JDBC method execution (see the discussion of fetchsize in the documentation for your JDBC driver). |
JAAS Configuration | String | This is not supported in Striim Cloud. | |
Password | encrypted password | The password for the specified user. See Encrypted passwords. | |
Parallel Threads | Integer | 1 | For Fast Snapshot Loading to Azure Synapse, BiqQuery, Databricks, Fabric Data Warehouse, Microsoft Dataverse, or Snowflake, specify the number of threads Database Reader will use. This value should not be higher than the number of tables to be read. For other targets, leave set to the default of 1. When Database Reader is started, the tables will be distributed among the threads in round-robin fashion, then each thread will read one table at a time. When the Tables property is a list of tables, the tables are read in that order. If the Tables property uses a wildcard, the sequence is determined by the order in which the JDBC driver gives Database Reader the list of tables. Known issue DEV-49013: the maximum number of tables supported by Parallel Threads is 255. |
Query | String | Optionally, specify a single SQL SELECT statement specifying the data to return. You may query tables, aliases, synonyms, and views. Query is not supported when Create Schema is True, Parallel Threads is greater than 1, or Restart Behavior on IL Interruption is Truncate_target_table or Replace_target_table. When If the query includes a synonym containing a period, it must be enclosed in escaped quotes. For example: If using a query when the output of a DatabaseReader source is the input of a DatabaseWriter target, specify the target table name as the value of DatabaseReader's Tables field. | |
Quiesce on IL Completion | Boolean | False | |
Restart Behavior on IL Interruption | Enum | Keep target table data | |
Return DateTime As | String |
| When the source is BigQuery or Snowflake, this setting is ignored and timestamp values are always returned as Java DateTime. Set to Set to If none of the above is the case, set to |
SSL Config | String | If the source is Oracle and it uses SSL, specify the required SSL properties (see the notes on SSL Config in Oracle Reader properties). | |
Tables | String | Specify the table(s) or view(s) to be read. MySQL, Oracle, and PostgreSQL names are case-sensitive, SQL Server names are not. Specify names as You may specify multiple tables and views as a list separated by semicolons or with the If you are using the Query property, specify QUERY as the table name. Modifying this property can interfere with recovery. If recovery is enabled for the application and it is necessary to modify the tables list, export the application (see Apps page), drop it, update the exported TQL with the new values, and import it (see Apps page). | |
Use Connection Profile | Boolean | False | See Introducing connection profiles. Appears in web UI only when Database Provider Type is BigQuery or Snowflake. |
Username | String | Specify the DBMS user name the adapter will use to log in to the server specified in ConnectionURL. For all databases, this user must have SELECT permission or privileges on the tables specified in the Tables property. For Db2 on z/OS, this user must also have SELECT privileges on the SYSCOLUMNS, SYSTABLES, and SYSVIEWS catalog tables. For Oracle, this user must also have SELECT privileges on DBA_TAB_COLS and ALL_COLL_TYPES. | |
Vendor Configuration | Striing | When reading from an Oracle PDB database, specify the PDB container name as follows, replacing {"Database" : { "Oracle" : { "CONTAINER" : "<PDB name>"}}} |
The output type is WAevent.
Note
To read from tables in both Oracle CDB and PDB databases, you must create two instances of DatabaseReader, one for each.
Sample Database ReaderWAEvent
For the following row:
id first_name last_name phone street city state zip_code 1 Deborah Burks NULL 9273 Thorne AV Orchard Park NY 14127
The WAEvent would be similar to:
WAEvent{ data: [1,"Deborah","Burks",null,"9273 Thorne AV","Orchard Park","NY","14127"] metadata: {"TableName":"BikeStores.sales.customers","ColumnCount":8, "OperationName":"SELECT","OPERATION_TS":1681412863364} userdata: null before: null dataPresenceBitMap: "fwM=" beforePresenceBitMap: "AAA=" typeUUID: {"uuidstring":"01edda2e-77f7-9b21-83c2-8e859085da65"} };
The operation name for Database Reader WAEvents is always SELECT.
DatabaseReader data type support and correspondence
The information in these tables also applies to Incremental Batch Reader.
For all sources except BigQuery, Db2, and Snowflake:
JDBC column type | TQL type | notes |
---|---|---|
Types.ARRAY | java.lang.String | |
Types.BIGINT | java.lang.Long | |
Types.BIT | java.lang.Boolean | |
Types.CHAR | java.lang.String | |
Types.DATE | org.joda.time.LocalDate | |
Types.DECIMAL | java.lang.String | |
Types.DOUBLE | java.lang.Double | |
Types.FLOAT | java.lang.Double | |
Types.INTEGER | java.lang.Integer | |
Types.NUMERIC | java.lang.String | |
Types.REAL | java.lang.Float | |
Types.SMALLINT | java.lang.Short | |
Types.TIMESTAMP | org.joda.time.DateTime | |
Types.TINYINT | java.lang.Short | For MySQL, if the source tables contain columns of this type, append |
Types.VARCHARCHAR | java.lang.String | |
other types | java.lang.String |
DatabaseReader can not read Oracle RAW or LONG RAW columns (Oracle Reader can).
For BigQuery:
BigQuery type | TQL type |
---|---|
ARRAY | java.lang.String |
BIGNUMERIC | java.math.BigDecimal |
BOOL | java.lang.Boolean |
BYTES | java.lang.String |
DATE | java.time.LocalDate |
DATETIME | java.time.LocalDateTime |
FLOAT64 | java.lang.Double |
GEOGRAPHY | java.lang.String |
INT64 | java.lang.Long |
INTERVAL | java.lang.String |
JSON | java.lang.String |
NUMERIC | java.math.BigDecimal |
RANGE | java.lang.String |
STRING | java.lang.String |
STRUCT | java.lang.String |
TIME | java.lang.String |
TIMESTAMP | java.time.ZonedDateTime |
For Db2 for iSeries or LUW
DB2 type | Striim type |
---|---|
BIGINT | java.lang.String |
BINARY | java.lang.String |
BLOB | java.lang.String |
CHAR | java.lang.String |
CLOB | java.lang.String |
DATE | java.lang.String |
DBCLOB | java.lang.String |
DECFLOAT | java.lang.String |
DECIMAL | java.lang.String |
DOUBLE | java.lang.String |
GRAPHIC | java.lang.String |
INTEGER | java.lang.String |
NUMERIC | java.lang.String |
REAL | java.lang.String |
SMALLINT | java.lang.String |
TIME | java.lang.String |
TIMESTAMP | java.lang.String |
VARBINARY | java.lang.String |
VARCHAR | java.lang.String |
VARGRAPHIC | java.lang.String |
XML | java.lang.String |
For Db2 for z/OS
See Db2 for z/OS data type support and correspondence.
For Snowflake:
Snowflake type | Aliases | TQL type |
---|---|---|
ARRAY | - | java.lang.String |
BINARY | VARBINARY | java.lang.String |
DATE | - | java.time.LocalDate |
FLOAT | DOUBLE, DOUBLE PRECISION, FLOAT4, FLOAT8, REAL | java.lang.Double |
GEOGRAPHY | - | java.lang.String |
GEOMETRY | - | java.lang.String |
NUMBER | BIGINT, BYTEINT, DECIMAL, INT, INTEGER, NUMERIC, SMALLINT, TINYIN | java.math.BigDecimal |
OBJECT | - | java.lang.String |
TIME | - | java.time.String |
TIMESTAMP_LTZ | TIMESTAMP WITH LOCAL TIME ZONE, TIMESTAMPLTZ | java.time.ZonedDateTime |
TIMESTAMP_NTZ | DATETIME, TIMESTAMP, TIMESTAMP WITHOUT TIME ZONE, TIMESTAMPNTZ | java.time.LocalDateTime |
TIMESTAMP_TZ | TIMESTAMP WITH TIME ZONE, TIMESTAMPTZ | java.time.ZonedDateTime |
VARCHAR | CHAR VARYING, NCHAR VARYING, NVARCHAR, NVARCHAR2, STRING, TEXT CHAR, CHARACTER, NCHAR | java.lang.String |
VARIANT | - | java.lang.String |
Database Reader runtime considerations
Do not perform a REPLACE or ALTER ... RECOMPILE on an application or flow containing a Database Reader when recovery is enabled. If it is necessary to modify the application, export the application (see Apps page), drop it, update the exported TQL with the new values, and import it (see Apps page).
Runtime considerations when using Database Reader or Incremental Batch Reader with BigQuery sources
Known issue DEV-49033: If while reading from BigQuery tables Database Reader experiences back pressure for approximately six hours or more it might experience a session timeout exception or partial read of the table data, in which case the app can go into the Completed state without throwing any exceptions or giving any other indication that not all the data was read. (This can be verified via running a SELECT COUNT(*) query on the source table and comparing it with the the Rows Read metric of the respective table in Database Reader's Table Information monitoring metric.) This is due to BigQuery's Storage Read API having a maximum session timeout of six hours. To avoid this, try increasing the Parallel Threads property value of the target. On restart, set Database Reader's Restart Behavior on IL Interruption property to Replace target table or Truncate target table (see Fast Snapshot Recovery during initial load).
Known issue DEV-44030: having two or more datasets or tables with the same name but different cases is not supported.
To read partitioned tables that have
Require WHERE clause to query data
` enabled, include the parameterInsertPartitionFilter=true
in the connection URLBy default, fractional seconds will be returned with precision of only three digits (milliseconds). To return fractional seconds with precision of six digits (microseconds), include the parameters
Other=KeepRawStringDataTypes=TIME;UseStorageAPI=false
in the connection URL. SettingUseStorageAPI=false
will change the format of the strings returned forARRAY
types from[value1, value2, value3, ...]
to["value1", "value2", "value3", ...]
.By default, empty arrays and nulls are returned as
null
. To return both empty arrays and nulls as[]
, include the parameterOther=EmptyArraysAsNull=false
in the connection URL. Returning nulls asnull
and empty arrays as[]
is not supported.If you receive an exception when using the Query property with a valid query, try including the parameter
QueryPassthrough=true
in the connection URL.
Runtime considerations when using Database Reader with Db2 sources
Database Reader supports the following sources:
Db2 for iSeries (AS/400) V7R4M0
Db2 for LUW 11.5.9
Db2 for z/OS versions 12 and 13.
Before Striim can read from those sources, you must install the appropriate JDBC driver for your DB2 environment
For iSeries and LUW, install the appropriate JDBC driver for your DB2 environment (see Using a JDBC Connector to connect to DB2 on iSeries (AS400)) in
striim/lib
and restart Striim.For z/OS, install the
Db2jcc4.jar
driver andDb2jcc_license.jar
license file instriim/lib
and restart Striim. Contact IBM for these files. They may be available through your existing Db2 Connect license.
Using TLS with Db2 for z/OS
Transport layer security is supported for the TCP connection between Database Reader and IBM Db2 for z/OS. See IBM Data Server Driver for JDBC and SQLJ support for SSL for instructions.
Limitations for Db2 for iSeries and LUW
The following limitations do not apply to Db2 for z/OS.
Create Schema is not supported.
The schema conversion utility is not supported.
Incremental Batch Reader does not support reading from DB2 for iSeries or LUW and there is no CDC reader for DB2 for for iSeries or LUW. In other words, only "lift and shift" use cases are supported.
(Contact Striim support for more information.