Readers overview
The following is a summary of reader capabilities. For more information, see:
Sample applications for programmers for an introduction to WAEvent
WAEvent contents for change data. HP NonStop reader WAEvent fields, MySQL Reader WAEvent fields, Oracle Reader and OJet WAEvent fields, PostgreSQL Reader WAEvent fields, SQL Server readers WAEvent fields,
Parsers for discussion of many of the supported inputs
Avro Parser for discussion of AvroEvent
JSON Parser for discussion of user-defined JSON
SQL CDC replication examples and Replicating MongoDB data to Azure CosmosDB
How update and delete operations are handled in writers for discussion of "insert only"
Recovering applications for detailed information about recovery and its limitations
Readers summary table
reader | input(s) | output stream type(s) | supports replication | recoverable |
|---|---|---|---|---|
Cosmos DB Reader (see Azure Cosmos DB using Core (SQL) API) | Initial Load mode: Cosmos DB documents using Microsoft Azure Cosmos SDK for Azure CosmosDB SQL API Incremental mode: Cosmos DB documents using Cosmos DB's change feed | JSONNodeEvent | requires upsert support in writer, so only to Cosmos DB Writer and MongoDB Writer | yes, but see Cosmos DB Reader limitations |
JDBC from a supported DBMS (see Database Reader) | WAEvent | insert only | if output is persisted to a Kafka stream (or use Incremental Batch Reader instead) | |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), GoldenGate trail file, JSON, name-value pairs, Parquet, XML | AvroEvent (when input is Avro), user-defined JSON (when input is JSON), ParquetEvent (when input is Parquet) or WAEvent | for GoldenGate only | yes | |
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, Parquet, XML | user-defined JSON (when input is JSON), ParquetEvent (when input is Parquet) or WAEvent | no | yes | |
NonStop TMF audit trail | WAEvent | yes | yes | |
Apache access log, binary, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
JDBC from same sources as Database Reader | WAEvent | insert only | yes | |
Apache access log, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | yes | |
Java Management Extensions (JMX) | JSONNodeEvent | no | yes | |
Apache access log, Avro, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | yes | |
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | yes | |
MariaDB Reader (see MariaDB / SkySQL) | MariaDB Galera Cluster binary log (binlog) | WAEvent | yes | yes |
Mongo Cosmos DB Reader (see Azure Cosmos DB using Cosmos DB API for MongoDB) | Initial Load mode: Azure Cosmos DB documents using mongo-driver-sync Incremental mode: Cosmos DB documents using Azure Cosmos DB API for MongoDB's change stream | JSONNodeEvent | insert and delete only requires upsert support in writer, so only to Cosmos DB Writer and MongoDB Writer | |
MongoDB Reader (see MongoDB) | MongoDB replica set operations log (oplog.rs) | JSONNodeEvent | yes | yes |
Avro, delimited text, JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | yes | |
MS SQL Reader / MSJet (see SQL Server) | SQL Server transaction log | WAEvent | yes | yes |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent, AvroEvent (when input is Avro), or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
MySQL Reader (see MySQL) | MySQL binary log (binlog) | WAEvent | yes | yes |
an OPC-UA server | OPCUA Data Change Event | no | yes | |
OJet (see Oracle Database) | Oracle logs | WAEvent | yes | yes |
Oracle Reader (see Oracle Database) | Oracle logs | WAEvent | yes | yes |
PostgreSQL Reader (see PostgreSQL) | PostgreSQL logical replication slot | WAEvent | yes | yes |
Apache access log, Avro, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, Parquet, XML | AvroEvent (when input is Avro), user-defined JSON (when input is JSON) ParquetEvent (when input is Parquet), or WAEvent | no | yes | |
Force.com REST API | WAEvent | yes | yes | |
Salesforce platform event message subscription | WAEvent | insert only | yes | |
Salesforce Streaming API | WAEvent | yes | yes | |
ServiceNow tables | WAEvent | insert and update only | yes | |
SQL Server | see MS SQL Reader / MS Jet, above | |||
Apache access log, binary, delimited text, free-form text (using RegEx), JSON, name-value pairs, XML | WAEvent or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
Teradata | see Database Reader, above | |||
Apache access log, binary, collectd, delimited text, free-form text (using RegEx), JSON, Kafka stream, name-value pairs, NetFlow v5 or v9, SNMP, XML | WAEvent, CollectdEvent (when input is collectd), or user-defined JSON (when input is JSON) | no | if output is persisted to a Kafka stream | |
Windows Application, Security, or System event log | WindowsLogEvent | no | yes |