Skip to main content

Working with non-SQL CDC readers

This section discusses the common characteristics of Striim's non-SQL-based change data capture readers.

Parsing and manipulating the fields of JSONNodeEvent

See JSONNode functions.

Adding user-defined data to JSONNodeEvent streams

Use the PutUserData function in a CQ to add a field to the JSONNodeEvent USERDATA map. For examples of how to use USERDATA elements in TQL, see Modifying output using ColumnMap and the discussions of PartitionKey in Kafka Writer and S3 Writer.

The following example would add the sixth element (counting from zero) in the JSONNodeEvent userdata array to USERDATA as the field "city":

INSERT INTO EnrichedStream
SELECT putUserData(x, 'city', data[5])
FROM MongoDBSourceStream x;

To remove an element from the USERDATA map, use the removeUserData function (you may specify multiple elements, separated by commas):

CREATE CQ RemoveUserData
INSERT INTO EnrichedStream 
SELECT removeUserData(x, 'city')
FROM MongoDBSourceStream x;

To remove all elements from the USERDATA map, use the clearUserData function:

CREATE CQ ClearUserData
INSERT INTO EnrichedStream 
SELECT clearUserData(x)
FROM MongoDBSourceStream x;

Converting JSONNodeEvent output to a user-defined type

Using the application and data from MongoDBReader example application and output, the following CQ would convert the JSONNodeEvent data array values to a Striim type, which could then be consumed by any other Striim component.

INSERT INTO EmployeeStream
SELECT data.get("_id").toString() AS id,
  data.get("firstname").toString() AS firstname,
  data.get("lastname").toString() AS lastname,
  data.get("age").toString() AS age
FROM MongoDBStream;