Skip to main content

Accepting data from external sources using Kafka

This topic describes how to configure Striim Cloud to accept data from external systems through the Striim-managed Kafka cluster. This enables integration with external change data capture tools and other data sources that can write to Kafka.

Overview

Striim Cloud includes an internal Kafka cluster for stream persistence. With external Kafka access enabled, systems outside of Striim can write data directly to this Kafka cluster. Striim applications can then consume this data using CREATE EXTERNAL SOURCE and write it to any supported target.

This architecture supports scenarios where:

  • Change data capture occurs outside of Striim (for example, using third-party CDC tools).

  • External applications need to feed data into Striim pipelines.

  • Data from on-premises systems must be streamed to Striim Cloud through Kafka.

External access uses mutual TLS (mTLS) for secure communication. Striim Cloud also provides an optional Karapace Schema Registry for schema management when working with Avro-formatted data.

Prerequisites

  • A Striim Cloud service instance running Striim 5.2.x or later.

  • The persistent streams feature enabled for the service.

  • The IP address or CIDR range of the external system that will connect to Kafka.

  • The external system configured and ready to write data to Kafka.

  • A target database or system ready to receive the replicated data.

Enabling external Kafka access

Enable external access to allow systems outside of Striim Cloud to write data to the Kafka cluster.

  1. In the Striim Cloud console, navigate to your service.

  2. Select the Persistent Streams tab.

  3. Under Kafka Cluster for Stream Persistence, select Enable External Kafka Access.

  4. In the Allowed Source IP Addresses field, enter the IP addresses or CIDR ranges of the external systems that will connect to Kafka. Separate multiple entries with commas.

  5. Click Save.

After enabling external access, the console displays the following connection information:

  • Kafka Bootstrap Server: The address and port for connecting to Kafka (for example, 203.0.113.10:9094).

  • Schema Registry URL: The address and port for the Schema Registry, if enabled (for example, https://203.0.113.10:8081).

Record these values for use when configuring the external system and Striim application.

Enabling Schema Registry

The Schema Registry provides schema management for Avro-formatted data. Enable the Schema Registry if your external system writes Avro data to Kafka.

  1. In the Persistent Streams tab, locate the Schema Registry section.

  2. Select the option to attach and expose the Schema Registry.

  3. Click Save.

The Schema Registry URL appears in the connection information after provisioning completes. The Schema Registry uses the same certificates as Kafka for mTLS authentication.

Note

The Schema Registry supports Avro format only in this release.

Downloading and preparing certificates

External systems and Striim applications authenticate to Kafka using mTLS certificates. Download the certificate bundle and prepare the certificates for use.

Downloading the certificate bundle

  1. In the Persistent Streams tab, click Download under Client Keys and Certificates.

  2. Extract the downloaded ZIP file. The bundle contains three files:

    • ca.crt: The certificate authority certificate.

    • client.crt: The client certificate.

    • client.key: The client private key.

Warning

The ca.crt file can only be downloaded once. Store it securely. If you lose this file, contact Striim support to obtain a replacement. The client.crt and client.key files can be downloaded again if needed.

Converting certificates to JKS format

Striim applications require certificates in Java KeyStore (JKS) format. Use the following commands to convert the PEM certificates to JKS format.

Create a PKCS12 keystore from the client certificate and key:

openssl pkcs12 -export \
  -in client.crt \
  -inkey client.key \
  -out kafka.client.p12 \
  -name kafka-client \
  -password pass:changeit

Convert the PKCS12 keystore to JKS format:

keytool -importkeystore \
  -srckeystore kafka.client.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass changeit \
  -destkeystore kafka.client.keystore.jks \
  -deststoretype JKS \
  -deststorepass changeit

Create a truststore containing the CA certificate:

keytool -import \
  -file ca.crt \
  -alias kafka-ca \
  -keystore kafka.client.truststore.jks \
  -storepass changeit \
  -noprompt

Upload the resulting kafka.client.keystore.jks and kafka.client.truststore.jks files to Striim using Manage Striim > Upload Files.

Creating the Striim application

Create a Striim application that reads from the external source via Kafka and writes to your target system. The application consists of a persisted stream, an external source, and a target.

Application structure

The following example shows the TQL structure for an application that reads from an external source and writes to a database target:

CREATE OR REPLACE APPLICATION ExternalSourceApp;

-- Create a persisted stream using the pre-configured Kafka properties
CREATE STREAM SourceData OF Global.WAEvent
  PERSIST USING admin.MTLSCloudKafkaProperties;

-- Create the external source
CREATE EXTERNAL SOURCE ExternalCDCSource (
  connector: '<connector_type>',
  configFile: '<path_to_config_file>',
  dataFormat: '<data_format>',
  schemaRegistry: '<schema_registry_url>',
  SchemaRegistryConfig: '
    ssl.keystore.location=<path_to_keystore>,
    ssl.keystore.password=<keystore_password>,
    ssl.key.password=<key_password>,
    ssl.truststore.location=<path_to_truststore>,
    ssl.truststore.password=<truststore_password>'
) OUTPUT TO SourceData;

-- Create the target
CREATE TARGET DatabaseTarget USING DatabaseWriter (
  ConnectionURL: '<jdbc_connection_url>',
  Username: '<username>',
  Password: '<password>',
  Tables: '<source_table>,<target_table>',
  BatchPolicy: 'EventCount:1000,Interval:5s',
  CommitPolicy: 'EventCount:1000,Interval:5s'
) INPUT FROM SourceData;

END APPLICATION ExternalSourceApp;

Property descriptions

Property

Description

connector

The connector type for the external source (for example, db2, oracle, postgres).

configFile

The path to the connector configuration file.

dataFormat

The data format used by the external source (for example, AvroExternalDb2zOS).

schemaRegistry

The Schema Registry URL displayed in the Striim Cloud console.

SchemaRegistryConfig

SSL configuration for connecting to the Schema Registry. Specify the paths to the uploaded keystore and truststore files along with their passwords.

Note

The admin.MTLSCloudKafkaProperties property set is pre-configured with the client certificates for connecting to the Striim Cloud Kafka cluster. Use this property set for persisted streams in applications that consume external data.

Deploying and starting the application

After creating the application, deploy and start it:

DEPLOY APPLICATION ExternalSourceApp;
START APPLICATION ExternalSourceApp;

When the application starts, Striim creates a Kafka topic for the persisted stream. The topic name follows the format <account>_<service_id>_<stream_name>. Provide this topic name to the external system for configuration.

Configuring the external system

Configure your external system to write data to the Striim Cloud Kafka cluster. The specific configuration steps vary by system, but generally require the following information:

Setting

Value

Kafka broker address

The Kafka Bootstrap Server address from the Striim Cloud console.

Kafka topic

The topic name created when you started the Striim application.

Schema Registry URL

The Schema Registry URL from the Striim Cloud console (if using Avro).

Security protocol

SSL (mTLS)

SSL certificates

The ca.crt, client.crt, and client.key files from the downloaded certificate bundle.

Verify connectivity from the external system to the Kafka bootstrap server address and port before starting data replication:

telnet <kafka_bootstrap_ip> <port>

Verifying data flow

After configuring both the Striim application and the external system:

  1. Insert or modify data in the source system monitored by the external CDC tool.

  2. Verify that the Striim application shows incoming events in the Flow Designer or Monitor view.

  3. Confirm that the data appears in the target system.

Limitations

  • The Schema Registry supports Avro format only.

  • The ca.crt certificate can only be downloaded once. Contact support if you need a replacement.

  • Certificate rotation is not automated. You receive a notification before certificates expire.

  • External access requires IP whitelisting. Update the allowed IP addresses if your external system's IP address changes.