Configuring Kafka for persisted streams
Kafka streams may be persisted to Striim's internal Kafka server or to an external Kafka server.
Using Striim's internal Kafka server
Warning
On Windows, Zookeeper and Kafka do not shut down cleanly. (This is a well-known problem.) Before you restart Kafka, you must delete the files they leave in c:\tmp
. Alternatively, look on stackoverflow.com for instructions on running Zookeeper and Kafka as services on Windows, or run an external Kafka server in a Linux virtual machine.
The default property set for the internal Kafka server that is installed with Striim at Striim/Kafka
is Global.DefaultKafkaProperties
:
jmx.broker=localhost:9998, bootstrap.brokers=localhost:9092, zk.address=localhost:2181
To change properties in an existing property set, see ALTER PROPERTYSET.
If you installed Striim with the JAR installer as discussed in Install Striim Platform for evaluation purposes and enabled Kafka in the setup wizard, it will start automatically. If you did not enable Kafka during installation, you may do so by re-running the setup wizard in the Strim/bin
directory (WebConfig.exe
for Windows, WebConfig
for Mac, or WebConfig.sh
for Linux).
If you installed Striim from a DEB, RPM, TGZ, or ZIP package as discussed in Running Striim in Ubuntu, Running Striim in CentOS, or Running Striim as a process, start Kafka as follows:
Open a terminal.
Change to
Striim/Kafka
, and enterbin/zookeeper-server-start.sh config/zookeeper.properties
(this will start Zookeeper).Open another terminal.
Change to
Striim/Kafka
and enterJMX_PORT=9998 bin/kafka-server-start.sh config/server.properties
(this will start Kafka).
You can then persist Kafka streams using the default property set.
Using an external Kafka server
When using an external Kafka server, to handle Striim's maximum batch size the following entries in config/server.properties
must have at least these minimum values:
message.max.bytes = 43264200 replica.fetch.max.bytes = 43264200 socket.request.max.bytes=104857600
To support persisting streams to an external server, use the Tungsten console to create a custom Striim property set using the following syntax:
CREATE PROPERTYSET <name> ( bootstrap.brokers:'<bootstrap IP address>:<port>', jmx.broker:'<jmx IP address>:<port>'), zk.address:'<zookeeper IP address>:<port>', partitions:'<number of partitions to use>' kafkaversion:'{0.8|0.9|0.10|0.11|2.1}';
If not specified, partitions
defaults to 200.
To change properties in an existing property set, see ALTER PROPERTYSET.
Using Kafka SASL (Kerberos) authentication with SSL encryption
To use SASL authentication with SSL encryption, do the following:
Get the files
krb5.conf
,principal.keytab
,server.keystore.jks
, andserver.truststore.jks
from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to/etc/striim/kafkaconf
.In the same directory, create the file
jaas.conf
, including the following lines, adjusting the keyTab path and principal to match your environment:KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true doNotPrompt=true client=true keyTab="/etc/striim/kafkaconf/principal.keytab" principal="principal@example.com"; };
Add the following to Striim's Java environment:
JAVA_SYSTEM_PROPERTIES=" \ -Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \ -Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "
Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.
security.protocol=SASL_SSL, sasl.kerberos.service.name=kafka, ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks, ssl.truststore.password=password, ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks, ssl.keystore.password=password, ssl.key.password=password
Using Kafka SASL (Kerberos) authentication without SSL encryption
To use SASL authentication without SSL encryption, do the following:
Get the files
krb5.conf
andprincipal.keytab
from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to/etc/striim/kafkaconf
.In the same directory, create the file
jaas.conf
, including the following lines, adjusting the keyTab path and principal to match your environment:KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true doNotPrompt=true client=true keyTab="/etc/striim/kafkaconf/principal.keytab" principal="principal@example.com"; };
Add the following to Striim's Java environment:
JAVA_SYSTEM_PROPERTIES=" \ -Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \ -Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "
Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig . For KafkaConfig, replace the comma with a semicolon.
security.protocol=SASL_PLAINTEXT, sasl.kerberos.service.name=kafka
Using Kafka SSL encryption without SASL (Kerberos) authentication
To use SSL encryption without SASL authentication, do the following:
Get the files
server.truststore.jks
andserver.keystore.jks
from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to/etc/striim/kafkaconf
.Include the following properties in your Kafka stream property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.
security.protocol=SSL, ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks, ssl.truststore.password=password, ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks, ssl.keystore.password=password, ssl.key.password=password
Using Kafka without SASL (Kerberos) authentication or SSL encryption
To use neither SASL authentication nor SSL encryption, do not specify security.protocol
in the KafkaReader or KafkaWriter KafkaConfig or in your Kafka stream's property set.
Additional properties for Kafka streams
Use these properties only in Kafka stream property sets, not with KafkaReader or KafkaWriter. Use single quotes around the values.
Kafka property | default value | notes |
---|---|---|
partitions | | the maximum number of Kafka partitions to be used if the stream is partitioned; if the stream is not partitioned, only one partition is used and this value is ignored |
replication.factor |
| the number of replicas to keep for each event; if this is greater than the number of brokers, creation of a topic will fail |