New in version 1.6.0.

unstable

Inside Kyuubi, the Kyuubi server uses the ApplicationManager module to manage all applications launched by itself, including different kinds of Kyuubi engines and self-contained applications.

The ApplicationManager leverages methods provided by application operation implementations derived from org.apache.kyuubi.engine.ApplicationOperation to monitor the status of those applications and kill abnormal applications in case they get orphaned and may introduce more methods in the future.

An ApplicationOperation implementation is usually built upon clients or APIs provided by cluster managers, such as Hadoop YARN, Kubernetes, etc.

For now, Kyuubi has already supported several built-in application operations:

  • JpsApplicationOperation: an operation that can manage apps with a local process, e.g. a local mode spark application
  • YarnApplicationOperation: an operation that can manage apps with a Hadoop Yarn cluster, e.g. a spark on yarn application
  • KubernetesApplicationOperation: an operation that can manage apps with a k8s cluster, e.g. a spark on k8s application

Besides those built-in ones, Kyuubi also supports loading custom ApplicationOperation through the Java ServiceLoader (SPI) for extra cluster managers.

The rest of this article will show you the specifications and steps to build and enable a custom operation.

  1. trait ApplicationOperation {
  3.   /**
  4.    * Step for initializing the instance.
  5.    */
  6.   def initialize(conf: KyuubiConf): Unit
  8.   /**
  9.    * Step to clean up the instance
  10.    */
  11.   def stop(): Unit
  13.   /**
  14.    * Called before other method to do a quick skip
  15.    *
  16.    * @param clusterManager the underlying cluster manager or just local instance
  17.    */
  18.   def isSupported(clusterManager: Option[String]): Boolean
  20.   /**
  21.    * Kill the app/engine by the unique application tag
  22.    *
  23.    * @param tag the unique application tag for engine instance.
  24.    *            For example,
  25.    *            if the Hadoop Yarn is used, for spark applications,
  26.    *            the tag will be preset via spark.yarn.tags
  27.    * @return a message contains response describing how the kill process.
  28.    *
  29.    * @note For implementations, please suppress exceptions and always return KillResponse
  30.    */
  31.   def killApplicationByTag(tag: String): KillResponse
  33.   /**
  34.    * Get the engine/application status by the unique application tag
  35.    *
  36.    * @param tag the unique application tag for engine instance.
  37.    * @return [[ApplicationInfo]]
  38.    */
  39.   def getApplicationInfoByTag(tag: String): ApplicationInfo
  40. }

An ApplicationInfo is used to represented the application information, including application id, name, state, url address and error message.

  1. object ApplicationState extends Enumeration {
  2.   type ApplicationState = Value
  3.   val PENDING, RUNNING, FINISHED, KILLED, FAILED, ZOMBIE, NOT_FOUND, UNKNOWN = Value
  4. }
  6. case class ApplicationInfo(
  7.     id: String,
  8.     name: String,
  9.     state: ApplicationState,
  10.     url: Option[String] = None,
  11.     error: Option[String] = None)

For application state mapping, you can reference the implementation of yarn:

  1. def toApplicationState(state: YarnApplicationState): ApplicationState = state match {
  2.   case YarnApplicationState.NEW => ApplicationState.PENDING
  3.   case YarnApplicationState.NEW_SAVING => ApplicationState.PENDING
  4.   case YarnApplicationState.SUBMITTED => ApplicationState.PENDING
  5.   case YarnApplicationState.ACCEPTED => ApplicationState.PENDING
  6.   case YarnApplicationState.RUNNING => ApplicationState.RUNNING
  7.   case YarnApplicationState.FINISHED => ApplicationState.FINISHED
  8.   case YarnApplicationState.FAILED => ApplicationState.FAILED
  9.   case YarnApplicationState.KILLED => ApplicationState.KILLED
  10.   case _ =>
  11.     warn(s"The yarn driver state: $state is not supported, " +
  12.       "mark the application state as UNKNOWN.")
  13.     ApplicationState.UNKNOWN
  14. }

Build A Custom Application Operation

  • reference kyuubi-server

    1. <dependency>
    2.    <groupId>org.apache.kyuubi</groupId>
    3.    <artifactId>kyuubi-server_2.12</artifactId>
    4.    <version>1.5.2-incubating</version>
    5.    <scope>provided</scope>
    6. </dependency>

  • create a custom class which implements the org.apache.kyuubi.engine.ApplicationOperation.

  • create a directory META-INF.services and a file with org.apache.kyuubi.engine.ApplicationOperation:

    META-INF.services/org.apache.kyuubi.engine.ApplicationOperation

    then add your fully-qualified name of custom application operation into the file.

Enable Custom Application Operation

Kyuubi uses Java SPI to load the custom Application Operation

  • compile and put the jar into $KYUUBI_HOME/jars