AmazonDynamoDB source connector

Description

Read data from Amazon DynamoDB.

Key features

Options

name type required default value
url string yes -
region string yes -
access_key_id string yes -
secret_access_key string yes -
table string yes -
schema config yes -
common-options yes -
scan_item_limit false -
parallel_scan_threads false -

url [string]

The URL to read to Amazon Dynamodb.

region [string]

The region of Amazon Dynamodb.

accessKeyId [string]

The access id of Amazon DynamoDB.

secretAccessKey [string]

The access secret of Amazon DynamoDB.

table [string]

The table of Amazon DynamoDB.

schema [Config]

fields [config]

Amazon Dynamodb is a NOSQL database service of support keys-value storage and document data structure,there is no way to get the data type.Therefore, we must configure schema.

such as:

  1. schema {
  2.   fields {
  3.     id = int
  4.     key_aa = string
  5.     key_bb = string
  6.   }
  7. }

common options

Source Plugin common parameters, refer to Source Plugin for details

scan_item_limit

number of item each scan request should return

parallel_scan_threads

number of logical segments for parallel scan

Example

  1. Amazondynamodb {
  2.   url = "http://127.0.0.1:8000"
  3.   region = "us-east-1"
  4.   accessKeyId = "dummy-key"
  5.   secretAccessKey = "dummy-secret"
  6.   table = "TableName"
  7.   schema = {
  8.     fields {
  9.       artist = string
  10.       c_map = "map<string, array<int>>"
  11.       c_array = "array<int>"
  12.       c_string = string
  13.       c_boolean = boolean
  14.       c_tinyint = tinyint
  15.       c_smallint = smallint
  16.       c_int = int
  17.       c_bigint = bigint
  18.       c_float = float
  19.       c_double = double
  20.       c_decimal = "decimal(30, 8)"
  21.       c_null = "null"
  22.       c_bytes = bytes
  23.       c_date = date
  24.       c_timestamp = timestamp
  25.     }
  26.   }
  27. }

Changelog

next version

  • Add Amazon DynamoDB Source Connector
  • Add source split to Amazondynamodb Connectors