In this page, we explain how to get your Hudi job to store into Baidu BOS.

Baidu BOS configs

There are two configurations required for Hudi-BOS compatibility:

  • Adding Baidu BOS Credentials for Hudi
  • Adding required Jars to classpath

Baidu BOS Credentials

Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your BOS bucket name, replace fs.bos.endpoint with your bos endpoint, replace fs.bos.access.key with your bos key, replace fs.bos.secret.access.key with your bos secret key. Hudi should be able to read/write from the bucket.

  1. <property>
  2.   <name>fs.defaultFS</name>
  3.   <value>bos://bucketname/</value>
  4. </property>
  6. <property>
  7.   <name>fs.bos.endpoint</name>
  8.   <value>bos-endpoint-address</value>
  9.   <description>Baidu bos endpoint to connect to,for example : http://bj.bcebos.com</description>
  10. </property>
  12. <property>
  13.   <name>fs.bos.access.key</name>
  14.   <value>bos-key</value>
  15.   <description>Baidu access key</description>
  16. </property>
  18. <property>
  19.   <name>fs.bos.secret.access.key</name>
  20.   <value>bos-secret-key</value>
  21.   <description>Baidu secret key.</description>
  22. </property>
  24. <property>
  25.   <name>fs.bos.impl</name>
  26.   <value>org.apache.hadoop.fs.bos.BaiduBosFileSystem</value>
  27. </property>

Baidu bos Libs

Baidu hadoop libraries jars to add to our classpath

  • com.baidubce:bce-java-sdk:0.10.165
  • bos-hdfs-sdk-1.0.2-community.jar

You can download the bos-hdfs-sdk jar from here , and then unzip it.