Version `GLIBC_2.23' not found when using dl4j-cuda

I am using dl4j-cuda under the version M1.1 and I am running my application on an HPC with cuda version 11.1. I have got an error: ```
Caused by: java.lang.UnsatisfiedLinkError: /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/nativeblas/linux-x86_64/libjnind4jcuda.so: /lib64/libm.so.6: version `GLIBC_2.23’ not found (required by /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/nativeblas/linux-x86_64/libnd4jcuda.so)

I tried to install GLIBC 2.23 but it didn't work. My dependencies are: 

<dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>cuda-platform-redist</artifactId>
            <version>11.2-8.1-1.5.5</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-cuda-11.2</artifactId>
            <version>1.0.0-M1.1</version>
            <classifier>linux-x86_64-cudnn</classifier>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.backend}</artifactId>
            <version>${dl4j-master.version}</version>
        </dependency>
<dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-cuda-11.2</artifactId>
            <version>1.0.0-M1.1</version>
        </dependency>

I tried to fix it it has been a while but I didn't succeed. Does anyone knows how to fix it ? 
Thank you!

What OS are you running on for the workers? You need a newer glibc in order to run this. If you are running on a very old system you will not be able to run your workload. I can try to help you figure out a better way to do this but will need more to work with.

These are the features of my OS:
NAME=“CentOS Linux”
VERSION=“7 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“7”
PRETTY_NAME=“CentOS Linux 7 (Core)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:7”
HOME_URL=“https://www.centos.org/
BUG_REPORT_URL=“https://bugs.centos.org/

CENTOS_MANTISBT_PROJECT=“CentOS-7”
CENTOS_MANTISBT_PROJECT_VERSION=“7”
REDHAT_SUPPORT_PRODUCT=“centos”
REDHAT_SUPPORT_PRODUCT_VERSION=“7”

My current glibc version is 2.17

@Nour-Rekik yeah centos 7 is end of life at this point. You will need to upgrade to 8 in order to run anything.

Thank you so much for your reply. That’s unfortunate cause I already installed GLIBC_2.23 but I still have the same error.
I wanted to know as well if gpu/Cuda is supported by the version beta7? So that I can switch to beta7 since I didn’t have such a problem with beta7.

@Nour-Rekik it is but not that cuda version. That version is also very old and won’t be as performant. Could you give me a full stack trace? If you have the same error it means glibc isn’t being used. You may not have installed it on the workers where this is running. Please double check how you did that.

This is the full error:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.nd4j.jita.concurrency.CudaAffinityManager.getNumberOfDevices(CudaAffinityManager.java:136)
        at org.nd4j.jita.constant.ConstantProtector.purgeProtector(ConstantProtector.java:60)
        at org.nd4j.jita.constant.ConstantProtector.<init>(ConstantProtector.java:53)
        at org.nd4j.jita.constant.ConstantProtector.<clinit>(ConstantProtector.java:41)
        at org.nd4j.jita.constant.ProtectedCudaConstantHandler.<clinit>(ProtectedCudaConstantHandler.java:69)
        at org.nd4j.jita.constant.CudaConstantHandler.<clinit>(CudaConstantHandler.java:38)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
        at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5152)
        at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5093)
        at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:270)
        at org.datavec.image.loader.NativeImageLoader.transformImage(NativeImageLoader.java:670)
        at org.datavec.image.loader.NativeImageLoader.asMatrix(NativeImageLoader.java:593)
        at org.datavec.image.loader.NativeImageLoader.asMatrix(NativeImageLoader.java:281)
        at org.datavec.image.loader.NativeImageLoader.asMatrix(NativeImageLoader.java:256)
        at org.datavec.image.loader.NativeImageLoader.asMatrix(NativeImageLoader.java:250)
        at org.datavec.image.recordreader.BaseImageRecordReader.next(BaseImageRecordReader.java:247)
        at org.datavec.image.recordreader.BaseImageRecordReader.nextRecord(BaseImageRecordReader.java:511)
        at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.initializeUnderlying(RecordReaderDataSetIterator.java:194)
        at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:341)
        at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:421)
        at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:53)
        at com.examples.DeepLearningOnSpark.imageNet_image.streaming.NetworkRetrainingMain.entryPoint(NetworkRetrainingMain.java:55)
        at com.examples.DeepLearningOnSpark.imageNet_image.streaming.NetworkRetrainingMain.main(NetworkRetrainingMain.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: https://deeplearning4j.konduit.ai/nd4j/backend
        at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:116)
        at org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:37)
        ... 38 more
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcuda in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1718)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1328)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1132)
        at org.nd4j.nativeblas.Nd4jCuda.<clinit>(Nd4jCuda.java:10)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
        at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:88)
        ... 39 more
Caused by: java.lang.UnsatisfiedLinkError: /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/nativeblas/linux-x86_64/libjnind4jcuda.so: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/nativeblas/linux-x86_64/libnd4jcuda.so)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1668)

I installed GLIBC_2.23 and I export it like that:

export PATH=/scratch/ws/1/s4122485-glibc/lib_new/bin:$PATH

export LD_LIBRARY_PATH=/scratch/ws/1/s4122485-glibc/lib_new:$LD_LIBRARY_PATH
 

Since I am using spark I did the configuration source framework-configure.sh spark $SPARK_HOME/conf and in spark-env.sh I added
export LD_LIBRARY_PATH=/scratch/ws/1/s4122485-glibc/lib_new/:$LD_LIBRARY_PATH
then continue with the start-all.sh

when doing ldd --version it’s GLIBC_2.23