Dockerfile - Add hadoop native libraries

[davidonlaptop]

When starting any HDFS daemon, we get this warning:

15/09/14 21:28:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Install the hadoop native libraries as a separate RUN instruction.

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

[codingtony]

No need to compile the native libs.

I was able to compile hadoop and the native libs, along with another Dockerfile (locally), but since the native libs are already included in the binary package that reside in the gelog/hadoop image, I don't see the point of doing this.

The easy fix is to install the right Ubuntu packages and set HADOOP_OPTS properly in the Dockerfile.

The only lib that does not work properly is the native bzip2. However It looks from the link you provided that they were not able to compile with bzip2 either..

After applying the patch :

docker run  --rm gelog/hadoop hadoop checknative
15/09/22 19:49:57 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
15/09/22 19:49:57 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/libsnappy.so.1
lz4:     true revision:99
bzip2:   false 
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so

[davidonlaptop]

Great!

I don't know about the bzip2. Do you have this issue on your cluster?

[codingtony]

Nope...works on our cluster
Will try to find out why

bzip2:   true /lib64/libbz2.so.1

[codingtony]

Maybe the hadoop version has not been compiled with libbz2 ?

root@6d97b5aa5843:/# hadoop checknative
15/09/22 22:11:02 WARN bzip2.Bzip2Factory: Failed to load/initialize
native-bzip2 library system-native, will use pure-Java version
15/09/22 22:11:02 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/libsnappy.so.1
lz4:     true revision:99
bzip2:   false
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
root@6d97b5aa5843:/# ls /usr/lib/x86_64-linux-gnu/lib
Display all 466 possibilities? (y or n)
root@6d97b5aa5843:/# dpkg -S libbz2
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1.0
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1.0.4
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0/changelog.Debian.gz
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0/copyright
root@6d97b5aa5843:/# ldd /lib/x86_64-linux-gnu/libbz2.so.1
        linux-vdso.so.1 =>  (0x00007fffa7a71000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcdd2585000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcdd2b5a000)

[davidonlaptop]

Technically, hadoop native libraries have been installed, but a problem remains with the bzip2. See #29 for more details.