ClickHouse Writer programmer's reference

CDDL Action

String

Process

See Handling schema evolution.

Connection Profile Name

String

See Connection profiles.

Connection Retry Policy

String

initialRetryDelay=10s, retryDelayMultiplier=2, maxRetryDelay=1m, maxAttempts=10, totalTimeout=10m

With the default setting, if a connection attempt is unsuccessful, the adapter will try again in 10 seconds (InitialRetryDelay=10s). If the second attempt is unsuccessful, in 20 seconds it will try a third time (InitialRetryDelay=10s multiplied by retryDelayMultiplier=2). If that fails, the adapter will try again in 40 seconds (the previous retry interval 20s multiplied by 2). If connection attempts continue to fail, the the adapter will try again every 60 seconds (maxRetryDelay=1m) until a total of five connection attempts have been made (maxAttempts=5), after which the adapter will halt and log an exception.

The adapter will halt when either maxAttempts or totalTimeout is reached.

InitialRetryDelay, maxRetryDelay, and totalTimeout may be specified in milliseconds (ms), seconds (s, the default), or minutes (m).

If retryDelayMultiplier is set to 1, connection will be attempted on the fixed interval set by InitialRetryDelay.

To disable connection retry, set maxAttempts=0.

Negative values are not supported.

Delete Strategy

String

LightweightDelete

Applies when Mode is MERGE (MergeTree) or when Table Engine is CoalescingMergeTree.

Controls the SQL mechanism used to physically remove rows from ClickHouse during DELETE and PKUPDATE operations in MERGE mode. Two strategies are supported:

Both strategies run with mutations_sync = 1, meaning the writer waits for the operation to complete before processing the next batch.

Excluded Tables

String

If Tables uses a wildcard, data from any tables specified here will be omitted. Multiple table names (separated by semicolons) and wildcards may be used exactly as for Tables.

Ignorable Exception Code

String

The only accepted value is the literal string TABLE_NOT_FOUND (case-insensitive); any other value is rejected at compile time. When set, a TABLE_NOT_FOUND error for a target table is logged and the triggering event is discarded and acknowledged instead of halting the application; the missing table is remembered so later events for that table are also discarded without re-querying ClickHouse. Use with care: ignoring this error means events for a missing table are silently dropped rather than retried or surfaced.

Mode

String

APPENDONLY

Applies only when Table Engine is MergeTree. Determines how CDC operations are applied to the target.

Mode is intended for MergeTree only. If you set Mode explicitly for a different table engine (for example, Mode = APPENDONLY with Table Engine = ReplacingMergeTree, which requires Mode = MERGE) the application halts at startup with a configuration error.

Null Marker

String

NULL

The token ClickHouse Writer writes to represent a NULL value in the batched CSV data. Change this only if your data legitimately contains the literal string "NULL" as a non-null value.

Optimized Merge

Boolean

False

Applies only when Table Engine is MergeTree and Mode is MERGE.

Set to True only when target’s input stream is the output of an HP NonStop reader, MySQL Reader, or Oracle Reader source and the source events will include partial records, or when the source is Oracle Reader and the source table includes BLOB or CLOB columns.

For example, with Oracle Reader, when supplemental logging has not been enabled for all columns, partial records are sent for updates. When the source events will always include full records, leave this set to false.

When Optimized Merge is True, each primary key update is handled in a separate write operation. If the source has frequent primary key updates, this may lead to a decline in write performance compared with Optimized Merge = False.

Parallel Threads

Integer

1

Creates multiple instances of writer for better performance. See Creating multiple writer instances. Events are distributed by target table, and events for the same table are always routed to the same instance to preserve ordering.Creating multiple writer instances

If the writer targets only one table, setting this higher than 1 has no effect; see Scaling writes to a single table for an alternative.

Table Engine

String

MergeTree

The ClickHouse table engine used for all tables written by this target. See Choosing a table engine for supported values and guidance.

Tables

String

Maps source to target tables. It supports the following forms:

If an incoming event's source table is not matched by any rule, the event is discarded (and counted in the discard metric).

Supports the KeyColumn and ColumnMap sub-clauses (see Modifying output using ColumnMap and Defining relations between source and target using ColumnMap and KeyColumns).

Limitation: Unicode and other special characters in source or target table names, column names, or Key Column names are not reliably supported; avoid them in identifiers that ClickHouse Writer maps. Very long column names (near ClickHouse's identifier length limit) can also fail. Test any non-ASCII or unusually long identifiers against your ClickHouse version before deploying to production.

Target Table Definition

String

See Target Table Definition.

Upload Policy

String

eventcount:10000, interval:30s

The upload policy may include eventcount and/or interval (see Setting output names and rollover / upload policies for syntax). Cached data is written to the storage account every time any of the specified values is exceeded. With the default value, data will be written every five minutes or sooner if the cache contains 10,000 events. When the app is undeployed, all remaining data is written to the storage account.