JsonPath transform plugin

Description

Support use jsonpath select data

Options

name type required default value
Columns Array Yes

common options [string]

Transform plugin common parameters, please refer to Transform Plugin for details

fields[array]

option

name type required default value
src_field String Yes
dest_field String Yes
path String Yes
dest_type String No String

src_field

the json source field you want to parse

Support SeatunnelDateType

  • STRING
  • BYTES
  • ARRAY
  • MAP
  • ROW

dest_field

after use jsonpath output field

dest_type

the type of dest field

path

Jsonpath

Read Json Example

The data read from source is a table like this json:

  1. {
  2.   "data": {
  3.     "c_string": "this is a string",
  4.     "c_boolean": true,
  5.     "c_integer": 42,
  6.     "c_float": 3.14,
  7.     "c_double": 3.14,
  8.     "c_decimal": 10.55,
  9.     "c_date": "2023-10-29",
  10.     "c_datetime": "16:12:43.459",
  11.     "c_array":["item1", "item2", "item3"]
  12.   }
  13. }

Assuming we want to use JsonPath to extract properties.

  1. transform {
  2.   JsonPath {
  3.     source_table_name = "fake"
  4.     result_table_name = "fake1"
  5.     columns = [
  6.      {
  7.         "src_field" = "data"
  8.         "path" = "$.data.c_string"
  9.         "dest_field" = "c1_string"
  10.      },
  11.      {
  12.         "src_field" = "data"
  13.         "path" = "$.data.c_boolean"
  14.         "dest_field" = "c1_boolean"
  15.         "dest_type" = "boolean"
  16.      },
  17.      {
  18.         "src_field" = "data"
  19.         "path" = "$.data.c_integer"
  20.         "dest_field" = "c1_integer"
  21.         "dest_type" = "int"
  22.      },
  23.      {
  24.         "src_field" = "data"
  25.         "path" = "$.data.c_float"
  26.         "dest_field" = "c1_float"
  27.         "dest_type" = "float"
  28.      },
  29.      {
  30.         "src_field" = "data"
  31.         "path" = "$.data.c_double"
  32.         "dest_field" = "c1_double"
  33.         "dest_type" = "double"
  34.      },
  35.       {
  36.          "src_field" = "data"
  37.          "path" = "$.data.c_decimal"
  38.          "dest_field" = "c1_decimal"
  39.          "dest_type" = "decimal(4,2)"
  40.       },
  41.       {
  42.          "src_field" = "data"
  43.          "path" = "$.data.c_date"
  44.          "dest_field" = "c1_date"
  45.          "dest_type" = "date"
  46.       },
  47.       {
  48.          "src_field" = "data"
  49.          "path" = "$.data.c_datetime"
  50.          "dest_field" = "c1_datetime"
  51.          "dest_type" = "time"
  52.       },
  53.             {
  54.          "src_field" = "data"
  55.          "path" = "$.data.c_array"
  56.          "dest_field" = "c1_array"
  57.          "dest_type" = "array<string>"        
  58.       }
  59.     ]
  60.   }
  61. }

Then the data result table fake1 will like this

data c1_string c1_boolean c1_integer c1_float c1_double c1_decimal c1_date c1_datetime c1_array
too much content not to show this is a string true 42 3.14 3.14 10.55 2023-10-29 16:12:43.459 [“item1”, “item2”, “item3”]

Read SeatunnelRow Example

Suppose a column in a row of data is of type SeatunnelRow and that the name of the column is col

SeatunnelRow(col)other
nameage….
a18….

The JsonPath transform converts the values of seatunnel into an array,

  1. transform {
  2.   JsonPath {
  3.     source_table_name = "fake"
  4.     result_table_name = "fake1"
  5.     columns = [
  6.      {
  7.         "src_field" = "col"
  8.         "path" = "$[0]"
  9.         "dest_field" = "name"
  10.               "dest_type" = "string"
  11.      },
  12.         {
  13.         "src_field" = "col"
  14.         "path" = "$[1]"
  15.         "dest_field" = "age"
  16.               "dest_type" = "int"
  17.      }
  18.     ]
  19.   }
  20. }

Then the data result table fake1 will like this

name age col other
a 18 [“a”,18] …

Changelog

  • Add JsonPath Transform