Skip to main content

MapR Stream Writer

Writes to a stream in MapR Converged Data Platform version 5.1.

MapR Stream Writer properties

property

type

default value

notes

MapR Stream Config

String

Optionally specify Kafka producer properties, separated by semicolons.

Mode

String

Sync

See Setting KafkaWriter's mode property: sync versus async for discussion of this property.

Topic

String

Specify the fully-qualified topic name. The syntax is /<stream>:<topic>, where <stream> includes its path (which may be displayed using the command hadoop fs -ls /), for example, /tmp/myfirststream:topic1. The stream must already exist. Striim will create the topic if it does not exist.

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.Supported writer-formatter combinations

MapR Stream Writer sample application

CREATE TARGET MapRStreamSample USING MapRStreamWriter (
  MapRStreamConfig: "acks=0;batch.size=250000;max.request.size=40000000",
  Mode: "ASync",
  Topic:'/striim/myfirststream:topic1'
)
FORMAT USING JSONFormatter ()
INPUT FROM TypedCSVStream;

If the MapR topic is partitioned, events will be distributed among the partitions based on the target's input stream's PARTITION BY field. If the input stream is not partitioned, all events will be written to partition 0.

MapRStreamWriter is based on Kafka Writer and the KafkaWriter sample code can be used as a model by replacing the target with the sample code above and creating a MapR stream named striim. See Apache Kafka and MapR Streams: Terms, Techniques and New Designs for more information about MapR's Kafka implementation.