-
File Layouts
Configs The following describes the general file layout structure for Apache Hudi. Please refer the tech spec for a more detailed description of the file layouts. Hudi orga... -
Schedulers
Introduction Quartz Azkaban Oozie Launching Gobblin in Local Mode Example Config Files Uploading Files to HDFS Adding Gobblin jar Dependencies Launching the Job Launching ... -
Introduction
Documentation Apache Iceberg is an open table format for huge analytic datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala u... -
JDBC
Gobblin JDBC writer & publisher Proposed design Requirements Design summary Design detail WriterInitializer JdbcWriterInitializer AvroFieldsPickConverter AvroToJdbcEntryConve... -
DDL
Spark DDL To use Iceberg in Spark, first configure Spark catalogs . Iceberg uses Apache Spark’s DataSourceV2 API for data source and catalog implementations. CREATE TABLE Spark... -
Introduction to EventMesh
Introduction to EventMesh Features Introduction to EventMesh Apache EventMesh is a fully serverless platform used to build distributed event-driven applications. ... -
Distributed locks and synchronizers
Lock Redis or Valkey based distributed reentrant Lock object for Java and implements Lock interface. Uses pub/sub channel to notify other threads across all Redisson instances w... -
Distributed counters
Id generator Redis or Valkey based Java Id generator RIdGenerator generates unique numbers but not monotonically increased. At first request, batch of id numbers is allocated and... -
Client-Side Caching
Native implementation Client Side caching is implemented using client tracking listener through RESP3 protocol available in Redis or Valkey. It’s used to speed up read operation... -
Metrics for Gobblin ETL
Configuring Metrics and Event emission Operational Metrics Extractor Metrics Converter Metrics Fork Operator Metrics Row Level Policy Metrics Data Writer Metrics Runtime Eve...
