In this page, we explain how to get your Hudi spark job to store into Tencent Cloud COS.

Tencent Cloud COS configs

There are two configurations required for Hudi-COS compatibility:

  • Adding Tencent Cloud COS Credentials for Hudi
  • Adding required Jars to classpath

Tencent Cloud COS Credentials

Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your COS bucket name, replace fs.cosn.userinfo.secretId with your COS secret Id, replace fs.cosn.userinfo.secretKey with your COS key. Hudi should be able to read/write from the bucket.

  1.     <property>
  2.         <name>fs.defaultFS</name>
  3.         <value>cosn://bucketname</value>
  4.         <description>COS bucket name</description>
  5.     </property>
  7.     <property>
  8.         <name>fs.cosn.userinfo.secretId</name>
  9.         <value>cos-secretId</value>
  10.         <description>Tencent Cloud Secret Id</description>
  11.     </property>
  13.     <property>
  14.         <name>fs.cosn.userinfo.secretKey</name>
  15.         <value>cos-secretkey</value>
  16.         <description>Tencent Cloud Secret Key</description>
  17.     </property>
  19.     <property>
  20.         <name>fs.cosn.bucket.region</name>
  21.         <value>ap-region</value>
  22.         <description>The region where the bucket is located.</description>
  23.     </property>
  25.     <property>
  26.         <name>fs.cosn.bucket.endpoint_suffix</name>
  27.         <value>cos.endpoint.suffix</value>
  28.         <description>
  29.           COS endpoint to connect to. 
  30.           For public cloud users, it is recommended not to set this option, and only the correct area field is required.
  31.         </description>
  32.     </property>
  34.     <property>
  35.         <name>fs.cosn.impl</name>
  36.         <value>org.apache.hadoop.fs.CosFileSystem</value>
  37.         <description>The implementation class of the CosN Filesystem.</description>
  38.     </property>
  40.     <property>
  41.         <name>fs.AbstractFileSystem.cosn.impl</name>
  42.         <value>org.apache.hadoop.fs.CosN</value>
  43.         <description>The implementation class of the CosN AbstractFileSystem.</description>
  44.     </property>

Tencent Cloud COS Libs

COS hadoop libraries to add to our classpath

  • org.apache.hadoop:hadoop-cos:2.8.5