Salesforce Reader
Reads Salesforce sObject data. See also Salesforce Platform Event Reader and Salesforce Push Topic Reader.
See Getting Started with the Force.com REST API for information on which firewall ports must be open.
See API Request Limits and Allocations for information on Salesforce-side limits that may require you to limit how much and how quickly Salesforce Reader ingests data.
Salesforce Reader properties
property | type | default value | notes |
---|---|---|---|
API End Point | String | endpoint for your Force.com REST API | |
Auth Token | encrypted password | If autoAuthTokenRenewal is set to false , specify your Salesforce access token (see Set Up Authorization on developer.salesforce.com: the first section, "Setting Up OAuth 2.0," explains how to create a "connected app"; the second section, "Session ID Authorization," explains how to get the token using If autoAuthTokenRenewal is set to | |
Auto Auth Token Renewal | String | false | With the default value of False, when the specified Auth Token expires the application will halt and you will need to modify it to update the auth token before restarting. This setting is recommended only for development and testing, not in a production environment. Set to True to renew the auth token automatically. In this case, leave Auth Token blank and set the Consumer Key, Consumer Secret, Password, Security Token, and Username properties. |
Connection Retry Policy | String | retryInterval=30, maxRetries=3 | With the default setting, if a connection attempt is unsuccessful, the adapter will try again in 30 seconds ( When autoAuthTokenRenewal is false, this setting has no effect. |
Consumer Key | String | If autoAuthTokenRenewal is set to true, specify the Consumer Key (see Set Up Authorization on developer.salesforce.com). | |
Consumer Secret | encrypted password | If autoAuthTokenRenewal is set to true, specify the Consumer Secret (see Set Up Authorization on developer.salesforce.com). | |
Custom Objects Only | Boolean | False | By default, both standard and custom objects are included. Set to True to include only custom objects and exclude standard objects. |
Mode | String | InitialLoad | Use the default setting Set to Set to |
Objects | See sObjects. | ||
Password | encrypted password | If autoAuthTokenRenewal is set to true, specify the password for the specified username. | |
Polling Interval | String | 5 min | How often the data will be read, in minutes. We recommend the default value. The maximum value is 120. See API Request Limits and Allocations for more information on setting this value appropriately. |
Security Token | String | When autoAuthTokenRenewal is set to true, specify the security token for the specified username (see Reset Your Security Token on help.salesforce.com). | |
SObjects | String | | In the Flow Designer this property is shown as Objects. With the default wildcard value Alternatively, specify one or more objects to read, separating multiple objects with semicolons, for example, The objects must be queryable. The account associated with the specified Auth Token must have View All Data permission for the objects. |
Start Timestamp | String | By default, Salesforce Reader reads only new events. Optionally, specify the time (based on LastModifiedDate) from which to start reading older events in the format | |
Thread Pool Size | String | 5 | Set this to match your Salesforce Concurrent API Request Limit (see API Request Limits and Allocations). |
Username | String | If autoAuthTokenRenewal is set to true, specify an appropriate username (see Add a Single User on help.salesforce.com). |
The output type is WAEvent.
Salesforce Reader WAEvent example
WAEvent{ data: [ "a025j000004He03AAC", "AA", 100.0, "WA", "USD", null, null, null] metadata: { "LastModifiedDate":1646674721000, "TableName":"Business__c", "IsDeleted":false, "CustomObject":true, "OwnerId":"0055j000004c2D2AAI", "CreatedById":"0055j000004c2D2AAI", "OperationName":"INSERT", "CreatedDate":1646674721000, "attributes": "{type=Business__c, url=\/services\/data\/v51.0\/sobjects\/Business__c\/a025j000004He03AAC}", "LastModifiedById":"0055j000004c2D2AAI", "SystemModstamp":"2022-03-07T17:38:41.000Z"}, } userdata: null before: null dataPresenceBitMap: "HwA=" beforePresenceBitMap: "AAA=" typeUUID: {"uuidstring":"01ed7a69-eb41-3f71-8a71-8cae4cf129d6"} };
Salesforce Reader monitoring
The Salesforce reader monitors the following metrics:
Metric |
---|
Read timestamp (called as Last Event Modified Tiime) |
Number of deletes |
Number of inserts |
Number of updates |
CPU |
CPU rate |
CPU rate per node |
Number of servers |
Number of events seen per monitor snapshot interval |
Source input |
Source rate |
Input |
Input rate |
Rate |
Last event position |
Latest activity |
Read lag |
I/O latency in ms |
Table name |
Verifying Salesforce Reader configuration
Use the following cURL commands (see Using cURL in the REST Examples and curl.haxx.se) to verify your configuration and get information about available resources and sObjects.
Get an access token using the Salesforce login URL.
curl https://login.salesforce.com/services/oauth2/token -d "grant_type=password" \ -d "client_id=<your consumer key>" -d "client_secret=<your consumer secret>" \ -d "username=<your username>" -d "password=<your password>"
Using the access token returned by that command, test the REST API URL for your organization. The instance is typically the first part of the URL you see in your browser when logged into Salesforce, such as "mycompany" in mycompany.salesforce.com. Alternatively, ask your Salesforce technical administrator for access to a connected app. (For more information, see Understanding the Username-Password OAuth Authentication Flow.)
If you do not have a proxy server:
curl https://<your Salesforce instance>.salesforce.com/services/data/ \ -H 'Authorization: Bearer <token>'
If you have a proxy server (change the proxy server URL to match yours):
curl -x http://mycompany.proxy.server.com:8080/ \ https://<your Salesforce instance >.salesforce.com/services/data/ \ -H 'Authorization: Bearer <token>'
List available REST resources and sObjects (see List Available REST Resources and Get a List of Objects).
curl https://<your Salesforce instance>.salesforce.com/services/data/v41.0 \ -H 'Authorization: Bearer <token>' curl https://<your Salesforce instance>.salesforce.com/services/data/v41.0/sobjects \ -H 'Authorization: Bearer <token>'
For additional information, see Salesforce's REST API Developer Guide .
Salesforce Reader example
The following TQL will read from the Business__c
sObject and create an appropriate typed stream:
CREATE SOURCE SFPoller USING SalesforceReader ( sObjects: 'Business__c', authToken: '********', apiEndPoint: '<your organization's endpoint UEL>', mode: 'InitialLoad', autoAuthTokenRenewal: 'false' ) OUTPUT TO DataStream; CREATE TYPE OpStream_Type ( Id String KEY, Name String, POSDataCode__c String, Currency__c String ); CREATE STREAM OpStream OF OpStream_Type; CREATE CQ CQ1 INSERT INTO OpStream SELECT data[0],data[1],data[2],data[3] FROM DataStream;
Replicating Salesforce data to Oracle
The following TQL will load all data from the Salesforce Business_c
object to the Oracle table BUSINESS_ORACLE_P
, then stop writing (in other words, new data will be ignored). Replace ********
with your Salesforce access token (see Understanding the Web Server OAuth Authentication Flow on developer.salesforce.com).
CREATE SOURCE SalesforceCloud USING SalesForceReader ( sObjects: 'Business__c', authToken: '********', mode: 'BulkLoad', apiEndPoint: 'https://ap2.salesforce.com', autoAuthTokenRenewal: false ) OUTPUT TO DataStream; CREATE TARGET OracleOnPremise USING DatabaseWriter ( ConnectionURL:'jdbc:oracle:thin:@192.168.123.18:1521/XE', Username:'system', Password:'manager', Tables: 'Business__c,SYSTEM.BUSINESS_ORACLE_P' ) INPUT FROM DataStream;
The following TQL will replicate multiple objects:
CREATE SOURCE SalesforceCloud USING SalesForceReader ( sObjects: 'Business1__c;Business2__c', apiEndPoint: 'https://ap2.salesforce.com', mode: 'Incremental', authToken: '***' ) OUTPUT TO DataStream; CREATE TARGET OracleOnPremise USING DatabaseWriter ( ConnectionURL: 'jdbc:oracle:thin:@localhost:1521/XE', Username: 'qatest', Tables: 'Business1__c,qatest.BUSINESS_ORACLE_P1 COLUMNMAP(ID=ID);Business2__c,qatest.BUSINESS_ORACLE_P2 COLUMNMAP(ID=ID)', Password: '***' INPUT FROM DataStream;
The following TQL will replicate new data continuously:
CREATE SOURCE SalesforceCloud USING SalesForceReader ( sObject: 'Business__c', authToken: '********', pollingInterval: '5 min', apiEndPoint: 'https://ap2.salesforce.com', mode: 'Incremental' ) OUTPUT TO DataStream; CREATE TARGET OracleOnPremise USING DatabaseWriter ( DriverName:'oracle.jdbc.OracleDriver', ConnectionURL:'jdbc:oracle:thin:@192.168.123.18:1521/XE', Username:'system', Password:'manager', Tables: 'Business__c,SYSTEM.BUSINESS_ORACLE_P' ) INPUT FROM DataStream;
Salesforce Platform Event Reader
Reads Salesforce platform events (user-defined notifications) using a subscription model (see Delivering Custom Notifications with Platform Events).
This adapter is based on Salesforce Reader. The output type and data type support are the same. The properties are the same, except for the following:
Mode and Polling Interval: Since platform events use a subscription model, these properties are omitted. Platform events are received as they are published by Salesforce.
Event Name: Specify the name of the platform event to subscribe to. The account associated with the
authToken
must have View All Data permission for the platform event. Any fields added to the platform event while the reader is running will be ignored.Replay From (default value
Earliest
): With the default value, will read all platform events currently in the retention window. To read only new events, set toTip
. To start reading from a specific point in the retention window, set to a valid ReplayID value (see Platform Event Fields.sObjects: Omitted since this adapter reads platform events, not sObjects.
When the output of this adapter is the input for DatabaseWriter and other table-based targets, events are insert-only. (Since platform events are one-time notifications, it does not make sense to update or delete them.)
Recovery (see Recovering applications) is supported provided that Striim is restarted within 24 hours, the length of time Salesforce holds events in the retention window.
Salesforce Push Topic Reader
Reads Salesforce sObject data using a PushTopic subscription (see Salesforce's Streaming API Developer Guide).
This gives you the latest data sooner than SalesforceReader. Note, however, that Salesforce restricts use of PushTopics in various ways, including limiting how many events you can read in a day and how many PushTopics you can create. See the PushTopic section of Streaming API Allocations for details.
This adapter is based on Salesforce Reader. The output type and data type support are the same. The properties are the same, except for the following:
Mode and Polling Interval: Since this uses a subscription model, these properties are omitted. sObject data is received as it is published by Salesforce.
PushTopic: The PushTopic to subscribe to. You must also specify the sObject to be read. (To read multiple sObjects from a PushTopic, create a source for each.) The account associated with the
authToken
must have View All Data permissions for both the PushTopic and the sObject.Replay From (default value
Earliest
): With the default value, will read all events currently in the PushTopic retention window. To read only new events, set toTip
. To start reading from a specific point in the retention window, set to a valid ReplayID value (see Message Durability).SObject: the standard or custom object to read, for example,
Account
orBusiness_C
(the account associated with the authToken must have View All Data permission for the object)
Recovery (see Recovering applications) is supported provided that Striim is restarted within 24 hours, the length of time Salesforce holds events in the retention window.