Redis sink connector

Description

Used to write data to Redis.

Key features

• exactly-once

Options

host [string]

Redis host

port [int]

Redis port

key [string]

The value of key you want to write to redis.

For example, if you want to use value of a field from upstream data as key, you can assign it to the field name.

Upstream data is the following:

If you assign field name to code and data_type to key, two data will be written to redis:

1. 200 -> {code: 200, message: true, data: get success}
2. 500 -> {code: 500, message: false, data: internal error}

If you assign field name to value and data_type to key, only one data will be written to redis because value is not existed in upstream data’s fields:

1. value -> {code: 500, message: false, data: internal error}

Please see the data_type section for specific writing rules.

Of course, the format of the data written here I just take json as an example, the specific or user-configured format prevails.

data_type [string]

Redis data types, support key hash list set zset

• key

Each data from upstream will be updated to the configured key, which means the later data will overwrite the earlier data, and only the last data will be stored in the key.

• hash

Each data from upstream will be split according to the field and written to the hash key, also the data after will overwrite the data before.

• list

Each data from upstream will be added to the configured list key.

• set

Each data from upstream will be added to the configured set key.

• zset

Each data from upstream will be added to the configured zset key with a weight of 1. So the order of data in zset is based on the order of data consumption.

user [string]

redis authentication user, you need it when you connect to an encrypted cluster

auth [string]

Redis authentication password, you need it when you connect to an encrypted cluster

db_num [int]

Redis database index ID. It is connected to db 0 by default

mode [string]

redis mode, single or cluster, default is single

nodes [list]

redis nodes information, used in cluster mode, must like as the following format:

[“host1:port1”, “host2:port2”]

format [string]

The format of upstream data, now only support json, text will be supported later, default json.

When you assign format is json, for example:

Upstream data is the following:

Connector will generate data as the following and write it to redis:

{"code":  200, "data":  "get success", "success":  "true"}

expire [long]

Set redis expiration time, the unit is second. The default value is -1, keys do not automatically expire by default.

common options

Sink plugin common parameters, please refer to Sink Common Options for details

Example

simple:

Redis {
  host = localhost
  port = 6379
  key = age
  data_type = list
}

Changelog

2.2.0-beta 2022-09-26

• Add Redis Sink Connector

next version

• [Improve] Support redis cluster mode connection and user authentication 3188