REST API

SeaTunnel has a monitoring API that can be used to query status and statistics of running jobs, as well as recent completed jobs. The monitoring API is a REST-ful API that accepts HTTP requests and responds with JSON data.

Overview

The monitoring API is backed by a web server that runs as part of the node, each node member can provide rest api capability. By default, this server listens at port 5801, which can be configured in hazelcast.yaml like :

  1. network:
  2.     rest-api:
  3.       enabled: true
  4.       endpoint-groups:
  5.         CLUSTER_WRITE:
  6.           enabled: true
  7.         DATA:
  8.           enabled: true
  9.     join:
  10.       tcp-ip:
  11.         enabled: true
  12.         member-list:
  13.           - localhost
  14.     port:
  15.       auto-increment: true
  16.       port-count: 100
  17.       port: 5801

API reference

Returns an overview over all jobs and their current state.

GET /hazelcast/rest/maps/running-jobs

(Returns an overview over all jobs and their current state.)

Parameters

Responses

  1. [
  2.   {
  3.     "jobId": "",
  4.     "jobName": "",
  5.     "jobStatus": "",
  6.     "envOptions": {
  7.     },
  8.     "createTime": "",
  9.     "jobDag": {
  10.       "vertices": [
  11.       ],
  12.       "edges": [
  13.       ]
  14.     },
  15.     "pluginJarsUrls": [
  16.     ],
  17.     "isStartWithSavePoint": false,
  18.     "metrics": {
  19.       "sourceReceivedCount": "",
  20.       "sinkWriteCount": ""
  21.     }
  22.   }
  23. ]

Return details of a job.

GET /hazelcast/rest/maps/running-job/:jobId

(Return details of a job.)

Parameters

name type data type description
jobId required long job id

Responses

  1. {
  2.   "jobId": "",
  3.   "jobName": "",
  4.   "jobStatus": "",
  5.   "createTime": "",
  6.   "jobDag": {
  7.     "vertices": [
  8.     ],
  9.     "edges": [
  10.     ]
  11.   },
  12.   "metrics": {
  13.     "sourceReceivedCount": "",
  14.     "sinkWriteCount": ""
  15.   },
  16.   "finishedTime": "",
  17.   "errorMsg": null,
  18.   "envOptions": {
  19.   },
  20.   "pluginJarsUrls": [
  21.   ],
  22.   "isStartWithSavePoint": false
  23. }

jobId, jobName, jobStatus, createTime, jobDag, metrics always be returned. envOptions, pluginJarsUrls, isStartWithSavePoint will return when job is running. finishedTime, errorMsg will return when job is finished.

When we can’t get the job info, the response will be:

  1. {
  2.   "jobId" : ""
  3. }

Return all finished Jobs Info.

GET /hazelcast/rest/maps/finished-jobs/:state

(Return all finished Jobs Info.)

Parameters

name type data type description
state optional string finished job status. FINISHED,CANCELED,FAILED,UNKNOWABLE

Responses

  1. [
  2.   {
  3.     "jobId": "",
  4.     "jobName": "",
  5.     "jobStatus": "",
  6.     "errorMsg": null,
  7.     "createTime": "",
  8.     "finishTime": "",
  9.     "jobDag": "",
  10.     "metrics": ""
  11.   }
  12. ]

Returns system monitoring information.

GET /hazelcast/rest/maps/system-monitoring-information

(Returns system monitoring information.)

Parameters

Responses

  1. [
  2.   {
  3.     "processors":"8",
  4.     "physical.memory.total":"16.0G",
  5.     "physical.memory.free":"16.3M",
  6.     "swap.space.total":"0",
  7.     "swap.space.free":"0",
  8.     "heap.memory.used":"135.7M",
  9.     "heap.memory.free":"440.8M",
  10.     "heap.memory.total":"576.5M",
  11.     "heap.memory.max":"3.6G",
  12.     "heap.memory.used/total":"23.54%",
  13.     "heap.memory.used/max":"3.73%",
  14.     "minor.gc.count":"6",
  15.     "minor.gc.time":"110ms",
  16.     "major.gc.count":"2",
  17.     "major.gc.time":"73ms",
  18.     "load.process":"24.78%",
  19.     "load.system":"60.00%",
  20.     "load.systemAverage":"2.07",
  21.     "thread.count":"117",
  22.     "thread.peakCount":"118",
  23.     "cluster.timeDiff":"0",
  24.     "event.q.size":"0",
  25.     "executor.q.async.size":"0",
  26.     "executor.q.client.size":"0",
  27.     "executor.q.client.query.size":"0",
  28.     "executor.q.client.blocking.size":"0",
  29.     "executor.q.query.size":"0",
  30.     "executor.q.scheduled.size":"0",
  31.     "executor.q.io.size":"0",
  32.     "executor.q.system.size":"0",
  33.     "executor.q.operations.size":"0",
  34.     "executor.q.priorityOperation.size":"0",
  35.     "operations.completed.count":"10",
  36.     "executor.q.mapLoad.size":"0",
  37.     "executor.q.mapLoadAllKeys.size":"0",
  38.     "executor.q.cluster.size":"0",
  39.     "executor.q.response.size":"0",
  40.     "operations.running.count":"0",
  41.     "operations.pending.invocations.percentage":"0.00%",
  42.     "operations.pending.invocations.count":"0",
  43.     "proxy.count":"8",
  44.     "clientEndpoint.count":"0",
  45.     "connection.active.count":"2",
  46.     "client.connection.count":"0",
  47.     "connection.count":"0"
  48.   }
  49. ]

Submit Job.

POST /hazelcast/rest/maps/submit-job

(Returns jobId and jobName if job submitted successfully.)

Parameters

name type data type description
jobId optional string job id
jobName optional string job name
isStartWithSavePoint optional string if job is started with save point

Body

  1. {
  2.     "env": {
  3.         "job.mode": "batch"
  4.     },
  5.     "source": [
  6.         {
  7.             "plugin_name": "FakeSource",
  8.             "result_table_name": "fake",
  9.             "row.num": 100,
  10.             "schema": {
  11.                 "fields": {
  12.                     "name": "string",
  13.                     "age": "int",
  14.                     "card": "int"
  15.                 }
  16.             }
  17.         }
  18.     ],
  19.     "transform": [
  20.     ],
  21.     "sink": [
  22.         {
  23.             "plugin_name": "Console",
  24.             "source_table_name": ["fake"]
  25.         }
  26.     ]
  27. }

Responses

  1. {
  2.     "jobId": 733584788375666689,
  3.     "jobName": "rest_api_test"
  4. }

Stop Job.

POST /hazelcast/rest/maps/stop-job

(Returns jobId if job stoped successfully.)

Body

  1. {
  2.     "jobId": 733584788375666689,
  3.     "isStopWithSavePoint": false # if job is stopped with save point
  4. }

Responses

  1. {
  2. "jobId": 733584788375666689
  3. }

Encrypt Config.

POST /hazelcast/rest/maps/encrypt-config

(Returns the encrypted config if config is encrypted successfully.)

For more information about customize encryption, please refer to the documentation config-encryption-decryption.

Body

  1. {
  2.     "env": {
  3.         "parallelism": 1,
  4.         "shade.identifier":"base64"
  5.     },
  6.     "source": [
  7.         {
  8.             "plugin_name": "MySQL-CDC",
  9.             "schema" : {
  10.                 "fields": {
  11.                     "name": "string",
  12.                     "age": "int"
  13.                 }
  14.             },
  15.             "result_table_name": "fake",
  16.             "parallelism": 1,
  17.             "hostname": "127.0.0.1",
  18.             "username": "seatunnel",
  19.             "password": "seatunnel_password",
  20.             "table-name": "inventory_vwyw0n"
  21.         }
  22.     ],
  23.     "transform": [
  24.     ],
  25.     "sink": [
  26.         {
  27.             "plugin_name": "Clickhouse",
  28.             "host": "localhost:8123",
  29.             "database": "default",
  30.             "table": "fake_all",
  31.             "username": "seatunnel",
  32.             "password": "seatunnel_password"
  33.         }
  34.     ]
  35. }

Responses

  1. {
  2.     "env": {
  3.         "parallelism": 1,
  4.         "shade.identifier": "base64"
  5.     },
  6.     "source": [
  7.         {
  8.             "plugin_name": "MySQL-CDC",
  9.             "schema": {
  10.                 "fields": {
  11.                     "name": "string",
  12.                     "age": "int"
  13.                 }
  14.             },
  15.             "result_table_name": "fake",
  16.             "parallelism": 1,
  17.             "hostname": "127.0.0.1",
  18.             "username": "c2VhdHVubmVs",
  19.             "password": "c2VhdHVubmVsX3Bhc3N3b3Jk",
  20.             "table-name": "inventory_vwyw0n"
  21.         }
  22.     ],
  23.     "transform": [],
  24.     "sink": [
  25.         {
  26.             "plugin_name": "Clickhouse",
  27.             "host": "localhost:8123",
  28.             "database": "default",
  29.             "table": "fake_all",
  30.             "username": "c2VhdHVubmVs",
  31.             "password": "c2VhdHVubmVsX3Bhc3N3b3Jk"
  32.         }
  33.     ]
  34. }