Use JuiceFS Hadoop Java SDK

JuiceFS provides Hadoop-compatible FileSystem by Hadoop Java SDK to support variety of components in Hadoop ecosystem.

NOTICE:

JuiceFS use local mapping of user and UID. So, you should sync all the needed users and their UIDs across the whole Hadoop cluster to avoid permission error.

Hadoop Compatibility

JuiceFS Hadoop Java SDK is compatible with Hadoop 2.x and Hadoop 3.x. As well as variety of components in Hadoop ecosystem.

In order to make JuiceFS works with other components, it usually takes 2 steps:

1. Put JAR file into the classpath of each Hadoop ecosystem component.
2. Put JuiceFS configurations into the configuration file of each Hadoop ecosystem component (usually core-site.xml).

Compiling

You need first installing Go 1.13+, JDK 8+ and Maven, then run following commands:

$ cd sdk/java
$ make

Deploy JuiceFS Hadoop Java SDK

After compiling you could find the JAR file in sdk/java/target directory, e.g. juicefs-hadoop-0.10.0.jar. Beware that file with original- prefix, it doesn't contain third-party dependencies. It's recommended to use the JAR file with third-party dependencies.

Note: The SDK could only be deployed to same operating system as it be compiled. For example, if you compile SDK in Linux then you must deploy it to Linux.

Then put the JAR file and $JAVA_HOME/lib/tools.jar to the classpath of each Hadoop ecosystem component. It's recommended create a symbolic link to the JAR file. The following tables describe where the SDK be placed.

Hadoop Distribution

Community Components

Configurations

Core Configurations

Cache Configurations

Others

When you use multiple JuiceFS file systems, all these configurations could be set to specific file system alone. You need put file system name in the middle of configuration, for example (replace {JFS_NAME} with appropriate value):

<property>
  <name>juicefs.{JFS_NAME}.meta</name>
  <value>redis://host:port/1</value>
</property>

Configurations Example

Note: Replace {HOST}, {PORT} and {DB} in juicefs.meta with appropriate values.

<property>
  <name>fs.jfs.impl</name>
  <value>io.juicefs.JuiceFileSystem</value>
</property>
<property>
  <name>fs.AbstractFileSystem.jfs.impl</name>
  <value>io.juicefs.JuiceFS</value>
</property>
<property>
  <name>juicefs.meta</name>
  <value>redis://{HOST}:{PORT}/{DB}</value>
</property>
<property>
  <name>juicefs.cache-dir</name>
  <value>/data*/jfs</value>
</property>
<property>
  <name>juicefs.cache-size</name>
  <value>1024</value>
</property>
<property>
  <name>juicefs.access-log</name>
  <value>/tmp/juicefs.access.log</value>
</property>

Configuration in Hadoop

Add configurations to core-site.xml.

CDH 6

Besides core-site, you also need to configure mapreduce.application.classpath of the YARN component, add:

$HADOOP_COMMON_HOME/lib/juicefs-hadoop.jar

HDP

Besides core-site, you also need to configure mapreduce.application.classpath of the MapReduce2 component, add (variables do not need to be replaced):

/usr/hdp/${hdp.version}/hadoop/lib/juicefs-hadoop.jar

Configuration in Flink

Add configurations to conf/flink-conf.yaml. You could only setup Flink client without modify configurations in Hadoop.

Restart Services

When the following components need to access JuiceFS, they should be restarted.

Note: Before restart, you need to confirm JuiceFS related configuration has been written to the configuration file of each component, usually you can find them in core-site.xml on the machine where the service of the component was deployed.

HDFS, Hue, ZooKeeper and other services don't need to be restarted.

When Class io.juicefs.JuiceFileSystem not found or No FilesSystem for scheme: jfs exceptions was occurred after restart, reference FAQ.

Verification

Hadoop

$ hadoop fs -ls jfs://{JFS_NAME}/

Hive

CREATE TABLE IF NOT EXISTS person
(
  name STRING,
  age INT
) LOCATION 'jfs://{JFS_NAME}/tmp/person';

Benchmark

JuiceFS provides some benchmark tools for you when JuiceFS has been deployed

Local environment

Meta

• create

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBenchWithoutMR -operation create -numberOfFiles 10000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench_local

  It creates 10000 empty files without write data

• open

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBenchWithoutMR -operation open -numberOfFiles 10000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench_local

  It opens 10000 files without read data

• rename

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBenchWithoutMR -operation rename -numberOfFiles 10000 -bytesPerBlock 134217728 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench_local

• delete

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBenchWithoutMR -operation delete -numberOfFiles 10000 -bytesPerBlock 134217728 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench_local

• for reference

IO Performance

• sequential write

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.TestFSIO -write -fileSize 20000 -baseDir jfs://{JFS_NAME}/benchmarks/fsio

• sequential read

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.TestFSIO -read -fileSize 20000 -baseDir jfs://{JFS_NAME}/benchmarks/fsio

  When run the cmd for the second time, the result may be much better than the first run. It's because the data was cached in memory, just clean the local disk cache.

• for reference

Distribute Benchmark

Distribute benchmark use MapReduce program to test meta and IO throughput performance

Enough resources should be provided to make sure all Map task can be started at the same time

We use 3 4c32g ecs(5Gbit/s) and AliYun Redis 5.0 4G redis for the benchmark

Meta

• create

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBench -operation create -threadsPerMap 10 -maps 10 -numberOfFiles 1000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench

  10 map task, each has 10 threads, each thread create 1000 empty file. 100000 files in total

• create

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBench -operation open -threadsPerMap 10 -maps 10 -numberOfFiles 1000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench

  10 map task, each has 10 threads, each thread open 1000 file. 100000 files in total

• create

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBench -operation rename -threadsPerMap 10 -maps 10 -numberOfFiles 1000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench

  10 map task, each has 10 threads, each thread rename 1000 file. 100000 files in total

• create

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.NNBench -operation delete -threadsPerMap 10 -maps 10 -numberOfFiles 1000 -baseDir jfs://{JFS_NAME}/benchmarks/nnbench

  10 map task, each has 10 threads, each thread delete 1000 file. 100000 files in total

• for reference

  • 10 threads

  operation	tps	delay(ms)
  create	2307	3.6
  open	3215	2.3
  rename	1700	5.22
  delete	1378	6.7

  • 100 threads

  operation	tps	delay(ms)
  create	8375	11.5
  open	12691	7.5
  rename	5343	18.4
  delete	3576	27.6

IO Performance

• sequential write

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.TestDFSIO -write -nrFiles 10 -fileSize 10000 -baseDir jfs://{JFS_NAME}/benchmarks/fsio

  10 map task, each task write 10000MB random data sequentially

• sequential read

  hadoop jar juicefs-hadoop.jar io.juicefs.bench.TestDFSIO -read -nrFiles 10 -fileSize 10000 -baseDir jfs://{JFS_NAME}/benchmarks/fsio

  10 map task, each task read 10000MB random data sequentially

• for reference

FAQ

Class io.juicefs.JuiceFileSystem not found exception

It means JAR file was not loaded, you can verify it by lsof -p {pid} | grep juicefs.

You should check whether the JAR file was located properly, or other users have the read permission.

Some Hadoop distribution also need to modify mapred-site.xml and put the JAR file location path to the end of the parameter mapreduce.application.classpath.

No FilesSystem for scheme: jfs exception

It means JuiceFS Hadoop Java SDK was not configured properly, you need to check whether there is JuiceFS related configuration in the core-site.xml of the component configuration.