Skip to main content

Using Private Service Connect with Google Cloud adapters

Google's Private Service Connect allows private services to be securely accessed from Virtual Private Cloud (VPC) networks without exposing the services to the public internet (for more information, see Virtual Private Cloud > Documentation > Guides > Private Service Connect). You can use Private Service Connect to access managed services across VPCs or to access Google APIs and services.

Typical use cases for Private Service Connect with Striim Cloud

In this release, Striim supports Private Service Connect for the following purposes:

  • For accessing Google APIs with the following adapters: BigQueryWriter, GCSReader, GCSWriter, and SpannerWriter.

  • For accessing Google APIs through an internal HTTPs load balancer. Allows you to connect regional Google APIs and services using an internal HTTP(S) load balancer in your VPC network without any external IP address. Supports only selected regional Google APIs and services (such as cloud key management, cloud logging, cloud run, and Pub/Sub).

  • For accessing published services (such as a VM instance serving as a database, cloud SQL, Snowflake, or Mongo DB Atlas). Supports connecting to published services in the same region but different VPCs without an external IP address.

Connecting to services securely using Private Service Connect

In a cloud-based infrastructure, services often communicate with each other over a public internet connection, making them vulnerable to various security threats. For example BigQuery Writer currently uses a publicly available API bigquery.googleapis.com to communicate and ingest data into BigQuery service.In the same way storage.googleapis.com and spanner.googleapis.com are used by GCS Writer and Spanner Writer respectively.

These API calls are routed through the internet before reaching the actual BigQuery instance or GCS bucket in your VPC.

Private Service Connect provides a secure way to connect services privately over the GCP network.

PSC_new_diagram_1.png
PSC_new_diagram_2.png

Preparing Striim Cloud to use Private Service Connect

Striim Cloud supports the following Private Service Connect endpoints:

  • Cloud SQL for MySQL

  • Cloud SQL for Postgres

  • Google Spanner

  • Google BigQuery

  • Google Cloud Storage

Note

MongoDB Atlas is not a currently supported endpoint.

You can perform the following prerequisites to support using Private Service Connect for these use cases:

  • Use Private Service Connect to access a published service

  • Use Private Service Connect to access Google APIs

Use Private Service Connect to access a published service

To access a published service:

  1. Publish your service and provide the service attachment URI while creating the private service connection.

    The service attachment URI will be in the following format:

    projects/<Service Project>/regions/<Region>/serviceAttachments/<Service Name>

    where,

    Service Project: Name of the project where the service is published.

    Region: Name of the region where the published service resides.

    Service Name: The published service name.

  2. Configure the approval method for the service attachment access though the private service connection.

    Approval may be automatic or manual. For the manual approval method, you can approve access whenever the private service connection is created for a project. While approving the connection, you can set the connection limit to that project. If the number of private service connections to that service attachment exceeds the limit, the next created private service connection to that service attachment goes into the pending state. You can accept the connection by increasing the connection limit.

  3. Create a Private Service Connect endpoint using a global internal IP address within the VPC. Note that Google Cloud Platform does not allow the use of special characters for the private endpoint with Google managed services.

  4. Assign a meaningful DNS name to the internal IP address used above.

    Note

    These names and IP addresses are internal to the VPC network and on-premises networks that are connected to it using Cloud VPN tunnels or VLAN attachments.

    Note

    DNS names will be automatically created for the Google managed services such as BigQuery, Storage, and Spanner. The DNS names are created with the following convention:

    <servicename>-<psc-endpointname>.p.googleapis.com

    For example:

    storage-striimdev.p.googleapis.com, spanner-striimqa.p.googleapis.com

  5. Provide the Private Service Connect IP address or DNS name to access the published service when configuring your pipeline in the Striim application. Whether to provide an IP address or a DNS name depends on the type of the service that you have published. Specify a value in one of the following formats for the Private Service Connect Endpoint property in the Striim adapter, so that the API calls are made using the private connection:

    • A PSC endpoint name as a string. For example,  striimdevpsc . The adapter will construct the full domain name for the specific service. This format is recommended for most users.

    • A full DNS name representing the PSC endpoint of specific service. This format is useful if you want to use a custom DNS name. For example, bigquery-striimdevpsc.p.googleapis.com, spanner-pscep2.p.googleapis.com, or mycustomdomainname.striimdns.com.

    For example, for Mongo DB Atlas, you need to provide the PSC name and IP address to configure the PSC in Mongo DB and access it. For a VM serving as a database, you can use the IP address or DNS name to access it. The DNS name format is pvtEpName.accShortName.installName.private-endpoint.

For specific procedures, see Connecting to VMs or databases in Google Cloud.

Use Private Service Connect to access Google APIs

To access Google APIs once you have created your Private Service Connect, you enter the PSC name in the template while creating your application. If your application is already running, you can undeploy the app, add the PSC parameter and restart the application.

Striim Cloud will construct the Google APIs in the format <service name>-<psc name>.p.googleapis.com. For example, storage-psctest.p.googleapis.com). Striim Cloud will use these API to access the service instead of the global Google APIs.

Sample application

The following sample application configure a Private Service Connect endpoint for a BigQuery Writer target:

CREATE APPLICATION OracleToBQ RECOVERY 10 SECOND INTERVAL;

CREATE OR REPLACE SOURCE oracle_source_CDC Using OracleReader(
 Username:’*****’,
 Password:’*****’,
 ConnectionURL:'jdbc:oracle:thin:@//localhost:1521/xe',
 OnlineCatalog:true,
 FetchSize:'1',
 Tables: 'HR.EMPLOYEE'
) Output To sourcestream1;


CREATE OR REPLACE TARGET bq_target USING BigQueryWriter  ( 
  projectId: 'striimdev'
  ,ServiceAccountKey: '/path/to/serviceaccountkey.json'
  ,StandardSQL: 'true'
  ,Mode: 'MERGE'
  ,optimizedMerge: 'true'
  ,PrivateServiceConnectEndpoint: 'striimdevpsc'
  ,BatchPolicy: 'eventCount:1000'
  ,Tables: 'HR.EMPLOYEE, HR.EMPLOYEE KeyColumns(RONUM)'
)   
INPUT FROM sourcestream1;

Usage notes

Note the following requirements and limitations for Private Service Connect support:

  • The Private Service Connect endpoint details you provide to the adapter must already exist. The adapter will not create the endpoint.

  • The Private Service Connect endpoint you provide to the adapter must be reachable or routable from the network where the Striim application is running. If the provided Private Service Connect endpoint becomes not reachable, the adapter will halt.

  • The BigQuery Storage Write API is currently not supported with Private Service Connect endpoints.