Iceberg support streaming and batch read With Apache Flink‘s DataStream API and Table API.

Reading with SQL

Iceberg support both streaming and batch read in Flink. Execute the following sql command to switch execution mode from streaming to batch, and vice versa:

  1. -- Execute the flink job in streaming mode for current session context
  2. SET execution.runtime-mode = streaming;
  3. -- Execute the flink job in batch mode for current session context
  4. SET execution.runtime-mode = batch;

Submit a Flink batch job using the following sentences:

  1. -- Execute the flink job in batch mode for current session context
  2. SET execution.runtime-mode = batch;
  3. SELECT * FROM sample;

Iceberg supports processing incremental data in Flink streaming jobs which starts from a historical snapshot-id:

  1. -- Submit the flink job in streaming mode for current session.
  2. SET execution.runtime-mode = streaming;
  3. -- Enable this switch because streaming read SQL will provide few job options in flink SQL hint options.
  4. SET table.dynamic-table-options.enabled=true;
  5. -- Read all the records from the iceberg current snapshot, and then read incremental data starting from that snapshot.
  6. SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/ ;
  7. -- Read all incremental data starting from the snapshot-id '3821550127947089987' (records from this snapshot will be excluded).
  8. SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s', 'start-snapshot-id'='3821550127947089987')*/ ;

There are some options that could be set in Flink SQL hint options for streaming job, see read options for details.

FLIP-27 source for SQL

Here are the SQL settings for the FLIP-27 source. All other SQL settings and options documented above are applicable to the FLIP-27 source.

  1. -- Opt in the FLIP-27 source. Default is false.
  2. SET table.exec.iceberg.use-flip27-source = true;

Reading branches and tags with SQL

Branch and tags can be read via SQL by specifying options. For more details refer to Flink Configuration

  1. --- Read from branch b1
  2. SELECT * FROM table /*+ OPTIONS('branch'='b1') */ ;
  3. --- Read from tag t1
  4. SELECT * FROM table /*+ OPTIONS('tag'='t1') */;
  5. --- Incremental scan from tag t1 to tag t2
  6. SELECT * FROM table /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s', 'start-tag'='t1', 'end-tag'='t2') */;

Reading with DataStream

Iceberg support streaming or batch read in Java API now.

Batch Read

This example will read all records from iceberg table and then print to the stdout console in flink batch job:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. DataStream<RowData> batch = FlinkSource.forRowData()
  4. .env(env)
  5. .tableLoader(tableLoader)
  6. .streaming(false)
  7. .build();
  8. // Print all records to stdout.
  9. batch.print();
  10. // Submit and execute this batch read job.
  11. env.execute("Test Iceberg Batch Read");

Streaming read

This example will read incremental records which start from snapshot-id ‘3821550127947089987’ and print to stdout console in flink streaming job:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. DataStream<RowData> stream = FlinkSource.forRowData()
  4. .env(env)
  5. .tableLoader(tableLoader)
  6. .streaming(true)
  7. .startSnapshotId(3821550127947089987L)
  8. .build();
  9. // Print all records to stdout.
  10. stream.print();
  11. // Submit and execute this streaming read job.
  12. env.execute("Test Iceberg Streaming Read");

There are other options that can be set, please see the FlinkSource#Builder.

Reading with DataStream (FLIP-27 source)

FLIP-27 source interface was introduced in Flink 1.12. It aims to solve several shortcomings of the old SourceFunction streaming source interface. It also unifies the source interfaces for both batch and streaming executions. Most source connectors (like Kafka, file) in Flink repo have migrated to the FLIP-27 interface. Flink is planning to deprecate the old SourceFunction interface in the near future.

A FLIP-27 based Flink IcebergSource is added in iceberg-flink module. The FLIP-27 IcebergSource is currently an experimental feature.

Batch Read

This example will read all records from iceberg table and then print to the stdout console in flink batch job:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. IcebergSource<RowData> source = IcebergSource.forRowData()
  4. .tableLoader(tableLoader)
  5. .assignerFactory(new SimpleSplitAssignerFactory())
  6. .build();
  7. DataStream<RowData> batch = env.fromSource(
  8. source,
  9. WatermarkStrategy.noWatermarks(),
  10. "My Iceberg Source",
  11. TypeInformation.of(RowData.class));
  12. // Print all records to stdout.
  13. batch.print();
  14. // Submit and execute this batch read job.
  15. env.execute("Test Iceberg Batch Read");

Streaming read

This example will start the streaming read from the latest table snapshot (inclusive). Every 60s, it polls Iceberg table to discover new append-only snapshots. CDC read is not supported yet.

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. IcebergSource source = IcebergSource.forRowData()
  4. .tableLoader(tableLoader)
  5. .assignerFactory(new SimpleSplitAssignerFactory())
  6. .streaming(true)
  7. .streamingStartingStrategy(StreamingStartingStrategy.INCREMENTAL_FROM_LATEST_SNAPSHOT)
  8. .monitorInterval(Duration.ofSeconds(60))
  9. .build();
  10. DataStream<RowData> stream = env.fromSource(
  11. source,
  12. WatermarkStrategy.noWatermarks(),
  13. "My Iceberg Source",
  14. TypeInformation.of(RowData.class));
  15. // Print all records to stdout.
  16. stream.print();
  17. // Submit and execute this streaming read job.
  18. env.execute("Test Iceberg Streaming Read");

There are other options that could be set by Java API, please see the IcebergSource#Builder.

Reading branches and tags with DataStream

Branches and tags can also be read via the DataStream API

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. // Read from branch
  4. DataStream<RowData> batch = FlinkSource.forRowData()
  5. .env(env)
  6. .tableLoader(tableLoader)
  7. .branch("test-branch")
  8. .streaming(false)
  9. .build();
  10. // Read from tag
  11. DataStream<RowData> batch = FlinkSource.forRowData()
  12. .env(env)
  13. .tableLoader(tableLoader)
  14. .tag("test-tag")
  15. .streaming(false)
  16. .build();
  17. // Streaming read from start-tag
  18. DataStream<RowData> batch = FlinkSource.forRowData()
  19. .env(env)
  20. .tableLoader(tableLoader)
  21. .streaming(true)
  22. .startTag("test-tag")
  23. .build();

Read as Avro GenericRecord

FLIP-27 Iceberg source provides AvroGenericRecordReaderFunction that converts Flink RowData Avro GenericRecord. You can use the convert to read from Iceberg table as Avro GenericRecord DataStream.

Please make sure flink-avro jar is included in the classpath. Also iceberg-flink-runtime shaded bundle jar can’t be used because the runtime jar shades the avro package. Please use non-shaded iceberg-flink jar instead.

  1. TableLoader tableLoader = ...;
  2. Table table;
  3. try (TableLoader loader = tableLoader) {
  4. loader.open();
  5. table = loader.loadTable();
  6. }
  7. AvroGenericRecordReaderFunction readerFunction = AvroGenericRecordReaderFunction.fromTable(table);
  8. IcebergSource<GenericRecord> source =
  9. IcebergSource.<GenericRecord>builder()
  10. .tableLoader(tableLoader)
  11. .readerFunction(readerFunction)
  12. .assignerFactory(new SimpleSplitAssignerFactory())
  13. ...
  14. .build();
  15. DataStream<Row> stream = env.fromSource(source, WatermarkStrategy.noWatermarks(),
  16. "Iceberg Source as Avro GenericRecord", new GenericRecordAvroTypeInfo(avroSchema));

Emitting watermarks

Emitting watermarks from the source itself could be beneficial for several purposes, like harnessing the Flink Watermark Alignment, or prevent triggering windows too early when reading multiple data files concurrently.

Enable watermark generation for an IcebergSource by setting the watermarkColumn. The supported column types are timestamp, timestamptz and long. Iceberg timestamp or timestamptz inherently contains the time precision. So there is no need to specify the time unit. But long type column doesn’t contain time unit information. Use
watermarkTimeUnit to configure the conversion for long columns.

The watermarks are generated based on column metrics stored for data files and emitted once per split. If multiple smaller files with different time ranges are combined into a single split, it can increase the out-of-orderliness and extra data buffering in the Flink state. The main purpose of watermark alignment is to reduce out-of-orderliness and excess data buffering in the Flink state. Hence it is recommended to set read.split.open-file-cost to a very large value to prevent combining multiple smaller files into a single split. The negative impact (of not combining small files into a single split) is on read throughput, especially if there are many small files. In typical stateful processing jobs, source read throughput is not the bottleneck. Hence this is probably a reasonable tradeoff.

This feature requires column-level min-max stats. Make sure stats are generated for the watermark column during write phase. By default, the column metrics are collected for the first 100 columns of the table. If watermark column doesn’t have stats enabled by default, use write properties starting with write.metadata.metrics when needed.

The following example could be useful if watermarks are used for windowing. The source reads Iceberg data files in order, using a timestamp column and emits watermarks:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. DataStream<RowData> stream =
  4. env.fromSource(
  5. IcebergSource.forRowData()
  6. .tableLoader(tableLoader)
  7. // Watermark using timestamp column
  8. .watermarkColumn("timestamp_column")
  9. .build(),
  10. // Watermarks are generated by the source, no need to generate it manually
  11. WatermarkStrategy.<RowData>noWatermarks()
  12. // Extract event timestamp from records
  13. .withTimestampAssigner((record, eventTime) -> record.getTimestamp(pos, precision).getMillisecond()),
  14. SOURCE_NAME,
  15. TypeInformation.of(RowData.class));

Example for reading Iceberg table using a long event column for watermark alignment:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
  2. TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
  3. DataStream<RowData> stream =
  4. env.fromSource(
  5. IcebergSource source = IcebergSource.forRowData()
  6. .tableLoader(tableLoader)
  7. // Disable combining multiple files to a single split
  8. .set(FlinkReadOptions.SPLIT_FILE_OPEN_COST, String.valueOf(TableProperties.SPLIT_SIZE_DEFAULT))
  9. // Watermark using long column
  10. .watermarkColumn("long_column")
  11. .watermarkTimeUnit(TimeUnit.MILLI_SCALE)
  12. .build(),
  13. // Watermarks are generated by the source, no need to generate it manually
  14. WatermarkStrategy.<RowData>noWatermarks()
  15. .withWatermarkAlignment(watermarkGroup, maxAllowedWatermarkDrift),
  16. SOURCE_NAME,
  17. TypeInformation.of(RowData.class));

Options

Read options

Flink read options are passed when configuring the Flink IcebergSource:

  1. IcebergSource.forRowData()
  2. .tableLoader(TableLoader.fromCatalog(...))
  3. .assignerFactory(new SimpleSplitAssignerFactory())
  4. .streaming(true)
  5. .streamingStartingStrategy(StreamingStartingStrategy.INCREMENTAL_FROM_LATEST_SNAPSHOT)
  6. .startSnapshotId(3821550127947089987L)
  7. .monitorInterval(Duration.ofMillis(10L)) // or .set("monitor-interval", "10s") \ set(FlinkReadOptions.MONITOR_INTERVAL, "10s")
  8. .build()

For Flink SQL, read options can be passed in via SQL hints like this:

  1. SELECT * FROM tableName /*+ OPTIONS('monitor-interval'='10s') */
  2. ...

Options can be passed in via Flink configuration, which will be applied to current session. Note that not all options support this mode.

  1. env.getConfig()
  2. .getConfiguration()
  3. .set(FlinkReadOptions.SPLIT_FILE_OPEN_COST_OPTION, 1000L);
  4. ...

Check out all the options here: read-options

Inspecting tables

To inspect a table’s history, snapshots, and other metadata, Iceberg supports metadata tables.

Metadata tables are identified by adding the metadata table name after the original table name. For example, history for db.table is read using db.table$history.

History

To show table history:

  1. SELECT * FROM prod.db.table$history;
made_current_at snapshot_id parent_id is_current_ancestor
2019-02-08 03:29:51.215 5781947118336215154 NULL true
2019-02-08 03:47:55.948 5179299526185056830 5781947118336215154 true
2019-02-09 16:24:30.13 296410040247533544 5179299526185056830 false
2019-02-09 16:32:47.336 2999875608062437330 5179299526185056830 true
2019-02-09 19:42:03.919 8924558786060583479 2999875608062437330 true
2019-02-09 19:49:16.343 6536733823181975045 8924558786060583479 true

This shows a commit that was rolled back. In this example, snapshot 296410040247533544 and 2999875608062437330 have the same parent snapshot 5179299526185056830. Snapshot 296410040247533544 was rolled back and is not an ancestor of the current table state.

Metadata Log Entries

To show table metadata log entries:

  1. SELECT * from prod.db.table$metadata_log_entries;
timestamp file latest_snapshot_id latest_schema_id latest_sequence_number
2022-07-28 10:43:52.93 s3://…/table/metadata/00000-9441e604-b3c2-498a-a45a-6320e8ab9006.metadata.json null null null
2022-07-28 10:43:57.487 s3://…/table/metadata/00001-f30823df-b745-4a0a-b293-7532e0c99986.metadata.json 170260833677645300 0 1
2022-07-28 10:43:58.25 s3://…/table/metadata/00002-2cc2837a-02dc-4687-acc1-b4d86ea486f4.metadata.json 958906493976709774 0 2

Snapshots

To show the valid snapshots for a table:

  1. SELECT * FROM prod.db.table$snapshots;
committed_at snapshot_id parent_id operation manifest_list summary
2019-02-08 03:29:51.215 57897183625154 null append s3://…/table/metadata/snap-57897183625154-1.avro { added-records -> 2478404, total-records -> 2478404, added-data-files -> 438, total-data-files -> 438, flink.job-id -> 2e274eecb503d85369fb390e8956c813 }

You can also join snapshots to table history. For example, this query will show table history, with the application ID that wrote each snapshot:

  1. select
  2. h.made_current_at,
  3. s.operation,
  4. h.snapshot_id,
  5. h.is_current_ancestor,
  6. s.summary['flink.job-id']
  7. from prod.db.table$history h
  8. join prod.db.table$snapshots s
  9. on h.snapshot_id = s.snapshot_id
  10. order by made_current_at;
made_current_at operation snapshot_id is_current_ancestor summary[flink.job-id]
2019-02-08 03:29:51.215 append 57897183625154 true 2e274eecb503d85369fb390e8956c813

Files

To show a table’s current data files:

  1. SELECT * FROM prod.db.table$files;
content file_path file_format spec_id partition record_count file_size_in_bytes column_sizes value_counts null_value_counts nan_value_counts lower_bounds upper_bounds key_metadata split_offsets equality_ids sort_order_id
0 s3:/…/table/data/00000-3-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet PARQUET 0 {1999-01-01, 01} 1 597 [1 -> 90, 2 -> 62] [1 -> 1, 2 -> 1] [1 -> 0, 2 -> 0] [] [1 -> , 2 -> c] [1 -> , 2 -> c] null [4] null null
0 s3:/…/table/data/00001-4-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet PARQUET 0 {1999-01-01, 02} 1 597 [1 -> 90, 2 -> 62] [1 -> 1, 2 -> 1] [1 -> 0, 2 -> 0] [] [1 -> , 2 -> b] [1 -> , 2 -> b] null [4] null null
0 s3:/…/table/data/00002-5-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet PARQUET 0 {1999-01-01, 03} 1 597 [1 -> 90, 2 -> 62] [1 -> 1, 2 -> 1] [1 -> 0, 2 -> 0] [] [1 -> , 2 -> a] [1 -> , 2 -> a] null [4] null null

Manifests

To show a table’s current file manifests:

  1. SELECT * FROM prod.db.table$manifests;
path length partition_spec_id added_snapshot_id added_data_files_count existing_data_files_count deleted_data_files_count partition_summaries
s3://…/table/metadata/45b5290b-ee61-4788-b324-b1e2735c0e10-m0.avro 4479 0 6668963634911763636 8 0 0 [[false,null,2019-05-13,2019-05-15]]

Note:

  1. Fields within partition_summaries column of the manifests table correspond to field_summary structs within manifest list, with the following order:
    • contains_null
    • contains_nan
    • lower_bound
    • upper_bound
  2. contains_nan could return null, which indicates that this information is not available from the file’s metadata. This usually occurs when reading from V1 table, where contains_nan is not populated.

Partitions

To show a table’s current partitions:

  1. SELECT * FROM prod.db.table$partitions;
partition spec_id record_count file_count total_data_file_size_in_bytes position_delete_record_count position_delete_file_count equality_delete_record_count equality_delete_file_count last_updated_at(μs) last_updated_snapshot_id
{20211001, 11} 0 1 1 100 2 1 0 0 1633086034192000 9205185327307503337
{20211002, 11} 0 4 3 500 1 1 0 0 1633172537358000 867027598972211003
{20211001, 10} 0 7 4 700 0 0 0 0 1633082598716000 3280122546965981531
{20211002, 10} 0 3 2 400 0 0 1 1 1633169159489000 6941468797545315876

Note: For unpartitioned tables, the partitions table will not contain the partition and spec_id fields.

All Metadata Tables

These tables are unions of the metadata tables specific to the current snapshot, and return metadata across all snapshots.

The “all” metadata tables may produce more than one row per data file or manifest file because metadata files may be part of more than one table snapshot.

All Data Files

To show all of the table’s data files and each file’s metadata:

  1. SELECT * FROM prod.db.table$all_data_files;
content file_path file_format partition record_count file_size_in_bytes column_sizes value_counts null_value_counts nan_value_counts lower_bounds upper_bounds key_metadata split_offsets equality_ids sort_order_id
0 s3://…/dt=20210102/00000-0-756e2512-49ae-45bb-aae3-c0ca475e7879-00001.parquet PARQUET {20210102} 14 2444 {1 -> 94, 2 -> 17} {1 -> 14, 2 -> 14} {1 -> 0, 2 -> 0} {} {1 -> 1, 2 -> 20210102} {1 -> 2, 2 -> 20210102} null [4] null 0
0 s3://…/dt=20210103/00000-0-26222098-032f-472b-8ea5-651a55b21210-00001.parquet PARQUET {20210103} 14 2444 {1 -> 94, 2 -> 17} {1 -> 14, 2 -> 14} {1 -> 0, 2 -> 0} {} {1 -> 1, 2 -> 20210103} {1 -> 3, 2 -> 20210103} null [4] null 0
0 s3://…/dt=20210104/00000-0-a3bb1927-88eb-4f1c-bc6e-19076b0d952e-00001.parquet PARQUET {20210104} 14 2444 {1 -> 94, 2 -> 17} {1 -> 14, 2 -> 14} {1 -> 0, 2 -> 0} {} {1 -> 1, 2 -> 20210104} {1 -> 3, 2 -> 20210104} null [4] null 0

All Manifests

To show all of the table’s manifest files:

  1. SELECT * FROM prod.db.table$all_manifests;
path length partition_spec_id added_snapshot_id added_data_files_count existing_data_files_count deleted_data_files_count partition_summaries
s3://…/metadata/a85f78c5-3222-4b37-b7e4-faf944425d48-m0.avro 6376 0 6272782676904868561 2 0 0 [{false, false, 20210101, 20210101}]

Note:

  1. Fields within partition_summaries column of the manifests table correspond to field_summary structs within manifest list, with the following order:
    • contains_null
    • contains_nan
    • lower_bound
    • upper_bound
  2. contains_nan could return null, which indicates that this information is not available from the file’s metadata. This usually occurs when reading from V1 table, where contains_nan is not populated.

References

To show a table’s known snapshot references:

  1. SELECT * FROM prod.db.table$refs;
name type snapshot_id max_reference_age_in_ms min_snapshots_to_keep max_snapshot_age_in_ms
main BRANCH 4686954189838128572 10 20 30
testTag TAG 4686954189838128572 10 null null