Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerfile - Add hadoop native libraries #8

Closed
davidonlaptop opened this issue Sep 15, 2015 · 5 comments
Closed

Dockerfile - Add hadoop native libraries #8

davidonlaptop opened this issue Sep 15, 2015 · 5 comments

Comments

@davidonlaptop
Copy link
Member

When starting any HDFS daemon, we get this warning:

15/09/14 21:28:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Install the hadoop native libraries as a separate RUN instruction.

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

@codingtony
Copy link
Contributor

No need to compile the native libs.

I was able to compile hadoop and the native libs, along with another Dockerfile (locally), but since the native libs are already included in the binary package that reside in the gelog/hadoop image, I don't see the point of doing this.

The easy fix is to install the right Ubuntu packages and set HADOOP_OPTS properly in the Dockerfile.

The only lib that does not work properly is the native bzip2. However It looks from the link you provided that they were not able to compile with bzip2 either..

After applying the patch :

docker run  --rm gelog/hadoop hadoop checknative
15/09/22 19:49:57 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
15/09/22 19:49:57 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/libsnappy.so.1
lz4:     true revision:99
bzip2:   false 
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so

codingtony pushed a commit to codingtony/docker-ubuntu-hadoop that referenced this issue Sep 22, 2015
@davidonlaptop
Copy link
Member Author

Great!

I don't know about the bzip2. Do you have this issue on your cluster?

@codingtony
Copy link
Contributor

Nope...works on our cluster
Will try to find out why

bzip2:   true /lib64/libbz2.so.1

@codingtony
Copy link
Contributor

Maybe the hadoop version has not been compiled with libbz2 ?

root@6d97b5aa5843:/# hadoop checknative
15/09/22 22:11:02 WARN bzip2.Bzip2Factory: Failed to load/initialize
native-bzip2 library system-native, will use pure-Java version
15/09/22 22:11:02 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/libsnappy.so.1
lz4:     true revision:99
bzip2:   false
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
root@6d97b5aa5843:/# ls /usr/lib/x86_64-linux-gnu/lib
Display all 466 possibilities? (y or n)
root@6d97b5aa5843:/# dpkg -S libbz2
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1.0
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1.0.4
libbz2-1.0:amd64: /lib/x86_64-linux-gnu/libbz2.so.1
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0/changelog.Debian.gz
libbz2-1.0:amd64: /usr/share/doc/libbz2-1.0/copyright
root@6d97b5aa5843:/# ldd /lib/x86_64-linux-gnu/libbz2.so.1
        linux-vdso.so.1 =>  (0x00007fffa7a71000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcdd2585000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcdd2b5a000)

@davidonlaptop
Copy link
Member Author

Technically, hadoop native libraries have been installed, but a problem remains with the bzip2. See #29 for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants