Using Private Service Connect with Google Cloud adapters

Google's Private Service Connect allows private services to be securely accessed from Virtual Private Cloud (VPC) networks without exposing the services to the public internet (for more information, see Virtual Private Cloud > Documentation > Guides > Private Service Connect). You can use Private Service Connect to access managed services across VPCs or to access Google APIs and services.

In this release, Striim supports Private Service Connect for BigQueryWriter, GCSReader, GCSWriter, and SpannerWriter.

Connecting to services securely using Private Service Connect

In a cloud-based infrastructure, services often communicate with each other over a public internet connection, making them vulnerable to various security threats. For example BigQuery Writer currently uses a publicly available API bigquery.googleapis.com to communicate and ingest data into BigQuery service.In the same way storage.googleapis.com and spanner.googleapis.com are used by GCS Writer and Spanner Writer respectively.

These API calls are routed through the internet before reaching the actual BigQuery instance or GCS bucket in your VPC.

Private Service Connect provides a secure way to connect services privately over the GCP network.

Using Private Service Connect with Striim adapters

Using Private Service Connect with Striim adapters involves the following steps:

Create a Private Service Connect endpoint using a global internal IP address within the VPC. Note that Google Cloud Platform does not allow the use of special characters for the private endpoint with Google managed services.
Assign a meaningful DNS name to the internal IP address used above.
Note
These names and IP addresses are internal to the VPC network and on-premises networks that are connected to it using Cloud VPN tunnels or VLAN attachments.
Note
DNS names will be automatically created for the Google managed services such as BigQuery, Storage, and Spanner. The DNS names are created with the following convention:
```
<servicename>-<psc-endpointname>.p.googleapis.com
```
For example:
```
storage-striimdev.p.googleapis.com, spanner-striimqa.p.googleapis.com
```
Specify a value in one of the following formats for the Private Service Connect Endpoint property in the Striim adapter, so that the API calls are made using the private connection:
- A PSC endpoint name as a string. For example, striimdevpsc. The adapter will construct the full domain name for the specific service. This format is recommended for most users.
- A full DNS name representing the PSC endpoint of specific service. This format is useful if you want to use a custom DNS name. For example, bigquery-striimdevpsc.p.googleapis.com, spanner-pscep2.p.googleapis.com, or mycustomdomainname.striimdns.com.

Sample application

The following sample application configure a Private Service Connect endpoint for a BigQuery Writer target:

CREATE APPLICATION OracleToBQ RECOVERY 10 SECOND INTERVAL;

CREATE OR REPLACE SOURCE oracle_source_CDC Using OracleReader(
 Username:’*****’,
 Password:’*****’,
 ConnectionURL:'jdbc:oracle:thin:@//localhost:1521/xe',
 OnlineCatalog:true,
 FetchSize:'1',
 Tables: 'HR.EMPLOYEE'
) Output To sourcestream1;


CREATE OR REPLACE TARGET bq_target USING BigQueryWriter  ( 
  projectId: 'striimdev'
  ,ServiceAccountKey: '/path/to/serviceaccountkey.json'
  ,StandardSQL: 'true'
  ,Mode: 'MERGE'
  ,optimizedMerge: 'true'
  ,PrivateServiceConnectEndpoint: 'striimdevpsc'
  ,BatchPolicy: 'eventCount:1000'
  ,Tables: 'HR.EMPLOYEE, HR.EMPLOYEE KeyColumns(RONUM)'
)   
INPUT FROM sourcestream1;

Usage notes

Note the following requirements and limitations for Private Service Connect support:

The Private Service Connect endpoint details you provide to the adapter must already exist. The adapter will not create the endpoint.
The Private Service Connect endpoint you provide to the adapter must be reachable or routable from the network where the Striim application is running. If the provided Private Service Connect endpoint becomes not reachable, the adapter will halt.
The BigQuery Storage Write API is currently not supported with Private Service Connect endpoints.