Notion source connector

Description

Used to read data from Notion.

Key features

Options

name type required default value
url String Yes -
password String Yes -
version String Yes -
method String No get
schema.fields Config No -
format String No json
params Map No -
body String No -
json_field Config No -
content_json String No -
poll_interval_millis int No -
retry int No -
retry_backoff_multiplier_ms int No 100
retry_backoff_max_ms int No 10000
enable_multi_lines boolean No false
common-options config No -

url [String]

http request url

password [String]

API key for login, you can get more detail at this link:

https://developers.notion.com/docs/authorization

version [String]

The Notion API is versioned. API versions are named for the date the version is released

method [String]

http request method, only supports GET, POST method

params [Map]

http params

body [String]

http body

poll_interval_millis [int]

request http api interval(millis) in stream mode

retry [int]

The max retry times if request http return to IOException

retry_backoff_multiplier_ms [int]

The retry-backoff times(millis) multiplier if request http failed

retry_backoff_max_ms [int]

The maximum retry-backoff times(millis) if request http failed

format [String]

the format of upstream data, now only support json text, default json.

when you assign format is json, you should also assign schema option, for example:

upstream data is the following:

  1. {
  2. "code": 200,
  3. "data": "get success",
  4. "success": true
  5. }

you should assign schema as the following:

  1. schema {
  2. fields {
  3. code = int
  4. data = string
  5. success = boolean
  6. }
  7. }

connector will generate data as the following:

code data success
200 get success true

when you assign format is text, connector will do nothing for upstream data, for example:

upstream data is the following:

  1. {
  2. "code": 200,
  3. "data": "get success",
  4. "success": true
  5. }

connector will generate data as the following:

content
{“code”: 200, “data”: “get success”, “success”: true}

schema [Config]

fields [Config]

the schema fields of upstream data

content_json [String]

This parameter can get some json data.If you only need the data in the ‘book’ section, configure content_field = "$.store.book.*".

If your return data looks something like this.

  1. {
  2. "store": {
  3. "book": [
  4. {
  5. "category": "reference",
  6. "author": "Nigel Rees",
  7. "title": "Sayings of the Century",
  8. "price": 8.95
  9. },
  10. {
  11. "category": "fiction",
  12. "author": "Evelyn Waugh",
  13. "title": "Sword of Honour",
  14. "price": 12.99
  15. }
  16. ],
  17. "bicycle": {
  18. "color": "red",
  19. "price": 19.95
  20. }
  21. },
  22. "expensive": 10
  23. }

You can configure content_field = "$.store.book.*" and the result returned looks like this:

  1. [
  2. {
  3. "category": "reference",
  4. "author": "Nigel Rees",
  5. "title": "Sayings of the Century",
  6. "price": 8.95
  7. },
  8. {
  9. "category": "fiction",
  10. "author": "Evelyn Waugh",
  11. "title": "Sword of Honour",
  12. "price": 12.99
  13. }
  14. ]

Then you can get the desired result with a simpler schema,like

  1. Http {
  2. url = "http://mockserver:1080/contentjson/mock"
  3. method = "GET"
  4. format = "json"
  5. content_field = "$.store.book.*"
  6. schema = {
  7. fields {
  8. category = string
  9. author = string
  10. title = string
  11. price = string
  12. }
  13. }
  14. }

Here is an example:

json_field [Config]

This parameter helps you configure the schema,so this parameter must be used with schema.

If your data looks something like this:

  1. {
  2. "store": {
  3. "book": [
  4. {
  5. "category": "reference",
  6. "author": "Nigel Rees",
  7. "title": "Sayings of the Century",
  8. "price": 8.95
  9. },
  10. {
  11. "category": "fiction",
  12. "author": "Evelyn Waugh",
  13. "title": "Sword of Honour",
  14. "price": 12.99
  15. }
  16. ],
  17. "bicycle": {
  18. "color": "red",
  19. "price": 19.95
  20. }
  21. },
  22. "expensive": 10
  23. }

You can get the contents of ‘book’ by configuring the task as follows:

  1. source {
  2. Http {
  3. url = "http://mockserver:1080/jsonpath/mock"
  4. method = "GET"
  5. format = "json"
  6. json_field = {
  7. category = "$.store.book[*].category"
  8. author = "$.store.book[*].author"
  9. title = "$.store.book[*].title"
  10. price = "$.store.book[*].price"
  11. }
  12. schema = {
  13. fields {
  14. category = string
  15. author = string
  16. title = string
  17. price = string
  18. }
  19. }
  20. }
  21. }

common options

Source plugin common parameters, please refer to Source Common Options for details

Example

  1. Notion {
  2. url = "https://api.notion.com/v1/users"
  3. password = "SeaTunnel-test"
  4. version = "2022-06-28"
  5. content_field = "$.results.*"
  6. schema = {
  7. fields {
  8. object = string
  9. id = string
  10. type = string
  11. person = {
  12. email = string
  13. }
  14. avatar_url = string
  15. }
  16. }
  17. }

Changelog

next version

  • Add Notion Source Connector