Skip to main content

Kafka Writer Initial Setup

Before you configure Kafka Writer, prepare the connection profiles, authentication credentials, encryption settings, and Schema Registry connectivity required to reach your Kafka cluster.

Review before you start

If you landed here directly, review these sections before creating the Kafka Writer application:

  • Key Concepts, because the steps below assume familiarity with Kafka topics, partitions, consumer groups, schema registries, and connection profiles.

  • Writing to AWS MSK, because Amazon MSK uses different broker addresses, authentication types, and setup steps than a self-managed Kafka cluster or Confluent Cloud.

  • Using the Striim Vault, because certificate-based authentication methods (Mutual TLS, SASL over SSL) require certificates and keys to be uploaded to the Vault before you configure a connection profile.

  • Connecting to Schema Registry, because the Avro Formatter and schema evolution depend on a Schema Registry connection profile being set up before you create the Kafka Writer application.

  • Required ACLs, because your Kafka administrator must grant these permissions to the Striim service account before Striim can connect.

  • Kafka Connection Profile Properties, for the full list of connection profile properties referenced throughout the steps below.

Supported Authentication Methods

Before configuring a connection profile, confirm your Kafka cluster and Schema Registry support one of the following authentication methods. Each is covered in detail in the setup steps below.

Kafka connection profiles support the following authentication methods:

  • AWS_MSK_IAM

  • GSSAPI

  • Mutual TLS

  • None

  • PLAIN

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Confluent Schema Registry connection profiles support the following authentication methods:

  • Basic

  • Confluent Cloud API

  • Mutual TLS

  • None

Setting up Connection Encryption

Striim allows a secure communication channel between Striim and the Kafka broker using encryption only, without any form of client authentication. This mode ensures data confidentiality over the network but does not enforce identity verification of the client. For Amazon MSK-specific guidance, see Setting up Connection Encryption.

This configuration is suitable when:

  • The data that is sent needs to be encrypted.

  • The Kafka broker does not require authenticating the client (e.g., test environments or internal trusted networks).

  • The Kafka broker is configured to allow one-way SSL.

Configure the Kafka Connection Profile with the following SSL-only settings:

Property Name

Value

Authentication Type

Choose NONE

Use SSL

True

Use Certificate

True (recommended) or False

CA Certificate

Path to PEM file (if Use Certificate is true)

SSL Truststore Location

Path to truststore file (if Use Certificate is false)

SSL Truststore Password

Password for truststore (if Use Certificate is false and truststore is password protected)

Note

To add encryption to SASL-based authentication, enable Use SSL in addition to configuring the JAAS Config for your chosen SASL mechanism. See Using SASL Authentication for details.

Using the Striim Vault

To securely reference configurations in a Kafka Connection Profile, you can store certificates and keys in the Vault.

Step 1: Create a New Vault

If you do not already have a vault, create one in Manage Striim > Vault.

Step 2: Upload Certificate Files

Upload the following files as FILE type in the vault:

  • CA Certificate File

  • SSL KeyStore Key

  • SSL KeyStore Certificate

Step 3: Configure the Kafka Connection Profile

Reference the uploaded Vault entries by entering the appropriate vault name and key names in the Kafka Connection Profile fields. If you are planning to use Avro Formatter, complete the setup of the required Schema Registry.