UnsatisfiedLinkError ( libjnind4jcpu.so, libopenblas_nolapack.so ) running native app on linux


I am trying to create a native app using DL4J and GraalVM. My objective is to create a distributable executable with all the dependencies included so that Users can just download and run it with installing JRE, gcc, etc.

I am using the 1.0.0-SNAPSHOT versions of ND4J and DL4J since they contain the 1.5.5 version of JavaCPP which has GraalVM support.

This are how my dependencies look like :

* Maven: org.nd4j:nd4j-api:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-common:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native:linux-x86_64:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native:macosx-x86_64:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native-api:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native-platform:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native-platform:linux-x86_64:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native-preset:1.0.0-SNAPSHOT
* Maven: org.nd4j:nd4j-native-preset:linux-x86_64:1.0.0-SNAPSHOT
* Maven: org.nd4j:protobuf:1.0.0-SNAPSHOT
* Maven: org.bytedeco:hdf5:linux-x86_64:1.12.0-1.5.5
* Maven: org.bytedeco:hdf5-platform:1.12.0-1.5.5
* Maven: org.bytedeco:javacpp:1.5.5
* Maven: org.bytedeco:javacpp:linux-x86_64:1.5.5
* Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.5
* Maven: org.bytedeco:javacpp-platform:1.5.5
* Maven: org.bytedeco:leptonica:linux-x86_64:1.80.0-1.5.5
* Maven: org.bytedeco:leptonica-platform:1.80.0-1.5.5
* Maven: org.bytedeco:mkl:2021.1-1.5.5
* Maven: org.bytedeco:mkl:linux-x86:2021.1-1.5.5
* Maven: org.bytedeco:mkl:linux-x86_64:2021.1-1.5.5
* Maven: org.bytedeco:mkl:macosx-x86_64:2021.1-1.5.5
* Maven: org.bytedeco:mkl:windows-x86:2021.1-1.5.5
* Maven: org.bytedeco:mkl:windows-x86_64:2021.1-1.5.5
* Maven: org.bytedeco:mkl-platform:2021.1-1.5.5
* Maven: org.bytedeco:openblas:0.3.13-1.5.5
* Maven: org.bytedeco:openblas:linux-x86_64:0.3.13-1.5.5
* Maven: org.bytedeco:openblas:macosx-x86_64:0.3.13-1.5.5
* Maven: org.bytedeco:openblas-platform:0.3.13-1.5.5
* Maven: org.bytedeco:opencv:linux-x86_64:4.5.1-1.5.5
* Maven: org.bytedeco:opencv-platform:4.5.1-1.5.5
* Maven: org.deeplearning4j:deeplearning4j-common:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-core:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-datasets:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-datavec-iterators:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-modelimport:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-nn:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-ui-components:1.0.0-SNAPSHOT
* Maven: org.deeplearning4j:deeplearning4j-utility-iterators:1.0.0-SNAPSHOT

This is how my resource config looks like :


I am able to build a native image using GraalVM for my project in a linux debian 8 docker image with libstdc+±8-dev installed. Following is the docker file :

RUN apt-get update && \
    apt-get install -y --force-yes gcc libz-dev upx libstdc++-8-dev

RUN wget https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-$VERSION/graalvm-ce-java11-linux-amd64-$VERSION.tar.gz && \
    tar -xvzf graalvm-ce-java11-linux-amd64-$VERSION.tar.gz && \
    rm graalvm-ce-java11-linux-amd64-$VERSION.tar.gz && \
    /graalvm-ce-java11-$VERSION/bin/gu install native-image
RUN wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz && \
    tar -xzvf apache-maven-3.3.9-bin.tar.gz -C /var/lib/ && \
    rm apache-maven-3.3.9-bin.tar.gz
RUN apt-get update && \
    apt-get install -y --force-yes build-essential
ARG RESULT_LIB="/var/lib/static"

RUN mkdir ${RESULT_LIB} && \
    curl -L -o musl.tar.gz https://musl.libc.org/releases/musl-1.2.1.tar.gz && \
    mkdir musl && tar -xvzf musl.tar.gz -C musl --strip-components 1 && cd musl && \
    ./configure --disable-shared --prefix=${RESULT_LIB} && \
    make && make install && \
    cd / && rm -rf /muscl && rm -f /musl.tar.gz && \
    cp /usr/lib/gcc/x86_64-linux-gnu/8/libstdc++.a ${RESULT_LIB}/lib/

ENV CC="musl-gcc"

RUN curl -L -o zlib.tar.gz https://zlib.net/zlib-1.2.11.tar.gz && \
   mkdir zlib && tar -xvzf zlib.tar.gz -C zlib --strip-components 1 && cd zlib && \
   ./configure --static --prefix=${RESULT_LIB} && \
    make && make install && \
    cd / && rm -rf /zlib && rm -f /zlib.tar.gz

ENV CC="musl-gcc"
ENV M2_HOME=/var/lib/apache-maven-3.3.9
ENV PATH=/graalvm-ce-java11-$VERSION/bin:/var/lib/apache-maven-3.3.9/bin:$PATH

However, when I try to execute the native binary on my production linux server without libstdc installed, I am getting UnsatisfiedLinkError

This is inspite of the file being present in the specified location and in the jar resources

root@04af621a9001:~/.javacpp/cache/linux-x86_64# ls -lah
total 269M
drwxr-xr-x 2 root root 4.0K Apr 23 19:16 .
drwxr-xr-x 3 root root 4.0K Apr 23 19:16 ..
lrwxrwxrwx 1 root root   13 Apr 23 19:16 libgcc_s.so -> libgcc_s.so.1
-rw-r--r-- 1 root root  89K Jan  1  1970 libgcc_s.so.1
lrwxrwxrwx 1 root root   16 Apr 23 19:16 libgfortran.so -> libgfortran.so.4
-rw-r--r-- 1 root root 1.5M Jan  1  1970 libgfortran.so.4
-rw-r--r-- 1 root root  55K Jan  1  1970 libjnijavacpp.so
-rw-r--r-- 1 root root 3.2M Jan  1  1970 libjnind4jcpu.so
-rw-r--r-- 1 root root  25M Jan  1  1970 libjniopenblas.so
-rw-r--r-- 1 root root 300K Jan  1  1970 libjniopenblas_nolapack.so
-rw-r--r-- 1 root root 181M Jan  1  1970 libnd4jcpu.so
lrwxrwxrwx 1 root root   16 Apr 23 19:16 libopenblas.so -> libopenblas.so.0
-rw-r--r-- 1 root root  30M Jan  1  1970 libopenblas.so.0
lrwxrwxrwx 1 root root   25 Apr 23 19:16 libopenblas_nolapack.so -> libopenblas_nolapack.so.0
-rw-r--r-- 1 root root  30M Jan  1  1970 libopenblas_nolapack.so.0
lrwxrwxrwx 1 root root   16 Apr 23 19:16 libquadmath.so -> libquadmath.so.0
-rw-r--r-- 1 root root 253K Jan  1  1970 libquadmath.so.0

Then I tried creating native image using --static flag with musl as libc ( details mentioned in the docker file ), assuming issues were due to CPP libraries not being present on the machine and native image relying on dynamic linking.
I was following this blog : GraalVM Native Image Tips & Tricks - James Ward
After doing this change, the application started giving the same error about libopenblas_nolapack.so library, again which is present in the given path

Error :
Caused by: java.lang.UnsatisfiedLinkError: Can’t load library: /root/.javacpp/cache/linux-x86_64/libjniopenblas_nolapack.so

So I wanted to know what I am missing. I am able to see the missing files in my fat jar, in my resource config, in my external dependencies so my thought is that it would not be due to some misconfigured pom.

@AdityaPawade can you DM me? I have a proof of concept with dl4j + graalvm, but it’s not ready for general use yet. It will involve using snapshots, not the current beta7 release though. If you’re fine with that, please let me know.

Hi @agibsonccc ,

Thanks for replying. Yes I am ok with using Snapshots.

Due to the following bug : [TensorFlow] No Java runtime present, requesting install · Issue #417 · bytedeco/javacpp · GitHub
I have already shifted to using snapshots since after moving to JavaCPP 1.5.5, there hasn’t been a beta launch.
I am using 1.0.0-SNAPSHOT version of DL4J and ND4J and 1.5.5 version of JavaCPP

Please let me know what can be done. Sorry could not find the option to DM