FakeSource connector

Support Those Engines

Spark
Flink
SeaTunnel Zeta

Description

The FakeSource is a virtual data source, which randomly generates the number of rows according to the data structure of the user-defined schema, just for some test cases such as type conversion or connector new feature testing

Key Features

Source Options

Name Type Required Default Description
tables_configs list no - Define Multiple FakeSource, each item can contains the whole fake source config description below
schema config yes - Define Schema information
rows config no - The row list of fake data output per degree of parallelism see title Options rows Case.
row.num int no 5 The total number of data generated per degree of parallelism
split.num int no 1 the number of splits generated by the enumerator for each degree of parallelism
split.read-interval long no 1 The interval(mills) between two split reads in a reader
map.size int no 5 The size of map type that connector generated
array.size int no 5 The size of array type that connector generated
bytes.length int no 5 The length of bytes type that connector generated
string.length int no 5 The length of string type that connector generated
string.fake.mode string no range The fake mode of generating string data, support range and template, default range,if use configured it to template, user should also configured string.template option
string.template list no - The template list of string type that connector generated, if user configured it, connector will randomly select an item from the template list
tinyint.fake.mode string no range The fake mode of generating tinyint data, support range and template, default range,if use configured it to template, user should also configured tinyint.template option
tinyint.min tinyint no 0 The min value of tinyint data that connector generated
tinyint.max tinyint no 127 The max value of tinyint data that connector generated
tinyint.template list no - The template list of tinyint type that connector generated, if user configured it, connector will randomly select an item from the template list
smallint.fake.mode string no range The fake mode of generating smallint data, support range and template, default range,if use configured it to template, user should also configured smallint.template option
smallint.min smallint no 0 The min value of smallint data that connector generated
smallint.max smallint no 32767 The max value of smallint data that connector generated
smallint.template list no - The template list of smallint type that connector generated, if user configured it, connector will randomly select an item from the template list
int.fake.template string no range The fake mode of generating int data, support range and template, default range,if use configured it to template, user should also configured int.template option
int.min int no 0 The min value of int data that connector generated
int.max int no 0x7fffffff The max value of int data that connector generated
int.template list no - The template list of int type that connector generated, if user configured it, connector will randomly select an item from the template list
bigint.fake.mode string no range The fake mode of generating bigint data, support range and template, default range,if use configured it to template, user should also configured bigint.template option
bigint.min bigint no 0 The min value of bigint data that connector generated
bigint.max bigint no 0x7fffffffffffffff The max value of bigint data that connector generated
bigint.template list no - The template list of bigint type that connector generated, if user configured it, connector will randomly select an item from the template list
float.fake.mode string no range The fake mode of generating float data, support range and template, default range,if use configured it to template, user should also configured float.template option
float.min float no 0 The min value of float data that connector generated
float.max float no 0x1.fffffeP+127 The max value of float data that connector generated
float.template list no - The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list
double.fake.mode string no range The fake mode of generating float data, support range and template, default range,if use configured it to template, user should also configured double.template option
double.min double no 0 The min value of double data that connector generated
double.max double no 0x1.fffffffffffffP+1023 The max value of double data that connector generated
double.template list no - The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list
common-options no - Source plugin common parameters, please refer to Source Common Options for details

Task Example

Simple:

This example Randomly generates data of a specified type. If you want to learn how to declare field types, click here.

  1. schema = {
  2. fields {
  3. c_map = "map<string, array<int>>"
  4. c_map_nest = "map<string, {c_int = int, c_string = string}>"
  5. c_array = "array<int>"
  6. c_string = string
  7. c_boolean = boolean
  8. c_tinyint = tinyint
  9. c_smallint = smallint
  10. c_int = int
  11. c_bigint = bigint
  12. c_float = float
  13. c_double = double
  14. c_decimal = "decimal(30, 8)"
  15. c_null = "null"
  16. c_bytes = bytes
  17. c_date = date
  18. c_timestamp = timestamp
  19. c_row = {
  20. c_map = "map<string, map<string, string>>"
  21. c_array = "array<int>"
  22. c_string = string
  23. c_boolean = boolean
  24. c_tinyint = tinyint
  25. c_smallint = smallint
  26. c_int = int
  27. c_bigint = bigint
  28. c_float = float
  29. c_double = double
  30. c_decimal = "decimal(30, 8)"
  31. c_null = "null"
  32. c_bytes = bytes
  33. c_date = date
  34. c_timestamp = timestamp
  35. }
  36. }
  37. }

Random Generation

16 data matching the type are randomly generated

  1. source {
  2. # This is a example input plugin **only for test and demonstrate the feature input plugin**
  3. FakeSource {
  4. row.num = 16
  5. schema = {
  6. fields {
  7. c_map = "map<string, string>"
  8. c_array = "array<int>"
  9. c_string = string
  10. c_boolean = boolean
  11. c_tinyint = tinyint
  12. c_smallint = smallint
  13. c_int = int
  14. c_bigint = bigint
  15. c_float = float
  16. c_double = double
  17. c_decimal = "decimal(30, 8)"
  18. c_null = "null"
  19. c_bytes = bytes
  20. c_date = date
  21. c_timestamp = timestamp
  22. }
  23. }
  24. result_table_name = "fake"
  25. }
  26. }

Customize the data content Simple:

This is a self-defining data source information, defining whether each piece of data is an add or delete modification operation, and defining what each field stores

  1. source {
  2. FakeSource {
  3. schema = {
  4. fields {
  5. c_map = "map<string, string>"
  6. c_array = "array<int>"
  7. c_string = string
  8. c_boolean = boolean
  9. c_tinyint = tinyint
  10. c_smallint = smallint
  11. c_int = int
  12. c_bigint = bigint
  13. c_float = float
  14. c_double = double
  15. c_decimal = "decimal(30, 8)"
  16. c_null = "null"
  17. c_bytes = bytes
  18. c_date = date
  19. c_timestamp = timestamp
  20. }
  21. }
  22. rows = [
  23. {
  24. kind = INSERT
  25. fields = [{"a": "b"}, [101], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"]
  26. }
  27. {
  28. kind = UPDATE_BEFORE
  29. fields = [{"a": "c"}, [102], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"]
  30. }
  31. {
  32. kind = UPDATE_AFTER
  33. fields = [{"a": "e"}, [103], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"]
  34. }
  35. {
  36. kind = DELETE
  37. fields = [{"a": "f"}, [104], "c_string", true, 117, 15987, 56387395, 7084913402530365000, 1.23, 1.23, "2924137191386439303744.39292216", null, "bWlJWmo=", "2023-04-22", "2023-04-22T23:20:58"]
  38. }
  39. ]
  40. }
  41. }

Due to the constraints of the HOCON specification, users cannot directly create byte sequence objects. FakeSource uses strings to assign bytes type values. In the example above, the bytes type field is assigned "bWlJWmo=", which is encoded from “miIZj” with base64. Hence, when assigning values to bytes type fields, please use strings encoded with base64.

Specified Data number Simple:

This case specifies the number of data generated and the length of the generated value

  1. FakeSource {
  2. row.num = 10
  3. map.size = 10
  4. array.size = 10
  5. bytes.length = 10
  6. string.length = 10
  7. schema = {
  8. fields {
  9. c_map = "map<string, array<int>>"
  10. c_array = "array<int>"
  11. c_string = string
  12. c_boolean = boolean
  13. c_tinyint = tinyint
  14. c_smallint = smallint
  15. c_int = int
  16. c_bigint = bigint
  17. c_float = float
  18. c_double = double
  19. c_decimal = "decimal(30, 8)"
  20. c_null = "null"
  21. c_bytes = bytes
  22. c_date = date
  23. c_timestamp = timestamp
  24. c_row = {
  25. c_map = "map<string, map<string, string>>"
  26. c_array = "array<int>"
  27. c_string = string
  28. c_boolean = boolean
  29. c_tinyint = tinyint
  30. c_smallint = smallint
  31. c_int = int
  32. c_bigint = bigint
  33. c_float = float
  34. c_double = double
  35. c_decimal = "decimal(30, 8)"
  36. c_null = "null"
  37. c_bytes = bytes
  38. c_date = date
  39. c_timestamp = timestamp
  40. }
  41. }
  42. }
  43. }

Template data Simple:

Randomly generated according to the specified template

Using template

  1. FakeSource {
  2. row.num = 5
  3. string.fake.mode = "template"
  4. string.template = ["tyrantlucifer", "hailin", "kris", "fanjia", "zongwen", "gaojun"]
  5. tinyint.fake.mode = "template"
  6. tinyint.template = [1, 2, 3, 4, 5, 6, 7, 8, 9]
  7. smalling.fake.mode = "template"
  8. smallint.template = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
  9. int.fake.mode = "template"
  10. int.template = [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
  11. bigint.fake.mode = "template"
  12. bigint.template = [30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
  13. float.fake.mode = "template"
  14. float.template = [40.0, 41.0, 42.0, 43.0]
  15. double.fake.mode = "template"
  16. double.template = [44.0, 45.0, 46.0, 47.0]
  17. schema {
  18. fields {
  19. c_string = string
  20. c_tinyint = tinyint
  21. c_smallint = smallint
  22. c_int = int
  23. c_bigint = bigint
  24. c_float = float
  25. c_double = double
  26. }
  27. }
  28. }

Range data Simple:

The specified data generation range is randomly generated

  1. FakeSource {
  2. row.num = 5
  3. string.template = ["tyrantlucifer", "hailin", "kris", "fanjia", "zongwen", "gaojun"]
  4. tinyint.min = 1
  5. tinyint.max = 9
  6. smallint.min = 10
  7. smallint.max = 19
  8. int.min = 20
  9. int.max = 29
  10. bigint.min = 30
  11. bigint.max = 39
  12. float.min = 40.0
  13. float.max = 43.0
  14. double.min = 44.0
  15. double.max = 47.0
  16. schema {
  17. fields {
  18. c_string = string
  19. c_tinyint = tinyint
  20. c_smallint = smallint
  21. c_int = int
  22. c_bigint = bigint
  23. c_float = float
  24. c_double = double
  25. }
  26. }
  27. }

Generate Multiple tables

This is a case of generating a multi-data source test.table1 and test.table2

  1. FakeSource {
  2. tables_configs = [
  3. {
  4. row.num = 16
  5. schema {
  6. table = "test.table1"
  7. fields {
  8. c_string = string
  9. c_tinyint = tinyint
  10. c_smallint = smallint
  11. c_int = int
  12. c_bigint = bigint
  13. c_float = float
  14. c_double = double
  15. }
  16. }
  17. },
  18. {
  19. row.num = 17
  20. schema {
  21. table = "test.table2"
  22. fields {
  23. c_string = string
  24. c_tinyint = tinyint
  25. c_smallint = smallint
  26. c_int = int
  27. c_bigint = bigint
  28. c_float = float
  29. c_double = double
  30. }
  31. }
  32. }
  33. ]
  34. }

Options rows Case

  1. rows = [
  2. {
  3. kind = INSERT
  4. fields = [1, "A", 100]
  5. },
  6. {
  7. kind = UPDATE_BEFORE
  8. fields = [1, "A", 100]
  9. },
  10. {
  11. kind = UPDATE_AFTER
  12. fields = [1, "A_1", 100]
  13. },
  14. {
  15. kind = DELETE
  16. fields = [1, "A_1", 100]
  17. }
  18. ]

Options table-names Case

  1. source {
  2. # This is a example source plugin **only for test and demonstrate the feature source plugin**
  3. FakeSource {
  4. table-names = ["test.table1", "test.table2", "test.table3"]
  5. parallelism = 1
  6. schema = {
  7. fields {
  8. name = "string"
  9. age = "int"
  10. }
  11. }
  12. }
  13. }

Changelog

2.2.0-beta 2022-09-26

  • Add FakeSource Source Connector

2.3.0-beta 2022-10-20

  • [Improve] Supports direct definition of data values(row) (2839)
  • [Improve] Improve fake source connector: (2944)
    • Support user-defined map size
    • Support user-defined array size
    • Support user-defined string length
    • Support user-defined bytes length
  • [Improve] Support multiple splits for fake source connector (2974)
  • [Improve] Supports setting the number of splits per parallelism and the reading interval between two splits (3098)

next version

  • [Feature] Support config fake data rows 3865
  • [Feature] Support config template or range for fake data 3932