Version `GLIBC_2.23' not found when using dl4j-cuda

Nour-Rekik · September 2, 2022, 5:04pm

I am getting this:

nore667e@taurusi8023:~/bigdl_pipeline> spark-submit --conf spark.driver.memory=400g --driver-java-options='-Dorg.bytedeco.javacpp.logger.debug=true'  --class com.examples.DeepLearningOnSpark.mnist_image.streaming_approach.MnistRetrainingMain  target/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar 
22/09/02 19:03:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.Loader
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/bytedeco/javacpp/linux-x86_64/libjnijavacpp.so
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.javacpp.Pointer
Warning: Versions of org.bytedeco:javacpp:1.5.7 and org.bytedeco:cuda:11.4-8.2-1.5.6 do not match.
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.cuda.global.cudart
Debug: Loading library cudart
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/bytedeco/cuda/linux-x86_64/libjnicudart.so
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.cuda.global.cudart
Debug: Loading class org.bytedeco.cuda.global.cublas
Debug: Loading class org.bytedeco.javacpp.presets.javacpp
Debug: Loading class org.bytedeco.cuda.global.cudart
Debug: Loading class org.bytedeco.cuda.global.cublas
Debug: Loading library cublasLt
Debug: Loading library cublas
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/bytedeco/cuda/linux-x86_64/libjnicublas.so
22/09/02 19:03:02 INFO Nd4jBackend: Loaded [JCublasBackend] backend
Debug: Loading class org.nd4j.linalg.jcublas.bindings.Nd4jCuda
Debug: Loading library curand
Debug: Loading library cusolver
Debug: Loading library cusparse
Debug: Loading library cudnn
Debug: Failed to load for cudnn@.8: java.lang.UnsatisfiedLinkError: no cudnn in java.library.path
Debug: Loading library cudnn_ops_infer
Debug: Failed to load for cudnn_ops_infer@.8: java.lang.UnsatisfiedLinkError: no cudnn_ops_infer in java.library.path
Debug: Loading library cudnn_ops_train
Debug: Failed to load for cudnn_ops_train@.8: java.lang.UnsatisfiedLinkError: no cudnn_ops_train in java.library.path
Debug: Loading library cudnn_adv_infer
Debug: Failed to load for cudnn_adv_infer@.8: java.lang.UnsatisfiedLinkError: no cudnn_adv_infer in java.library.path
Debug: Loading library cudnn_adv_train
Debug: Failed to load for cudnn_adv_train@.8: java.lang.UnsatisfiedLinkError: no cudnn_adv_train in java.library.path
Debug: Loading library cudnn_cnn_infer
Debug: Failed to load for cudnn_cnn_infer@.8: java.lang.UnsatisfiedLinkError: no cudnn_cnn_infer in java.library.path
Debug: Loading library cudnn_cnn_train
Debug: Failed to load for cudnn_cnn_train@.8: java.lang.UnsatisfiedLinkError: no cudnn_cnn_train in java.library.path
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/linalg/jcublas/bindings/linux-x86_64/libgomp.so.1
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/linalg/jcublas/bindings/linux-x86_64/libnd4jcuda.so
Debug: Loading /home/h4/nore667e/.javacpp/cache/deepLearningSimpleOne-1.0-SNAPSHOT-jar-with-dependencies.jar/org/nd4j/linalg/jcublas/bindings/linux-x86_64/libjnind4jcuda.so
22/09/02 19:03:03 INFO NativeOpsHolder: Number of threads used for linear algebra: 32
Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.examples.DeepLearningOnSpark.mnist_image.streaming_approach.MnistRetrainingMain.entryPoint(MnistRetrainingMain.java:27)
        at com.examples.DeepLearningOnSpark.mnist_image.streaming_approach.MnistRetrainingMain.main(MnistRetrainingMain.java:22)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
        at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5205)
        at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5070)
        at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:290)
        ... 14 more
Caused by: java.lang.NullPointerException
        at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5148)
        ... 16 more
22/09/02 19:03:03 INFO ShutdownHookManager: Shutdown hook called
22/09/02 19:03:03 INFO ShutdownHookManager: Deleting directory /tmp/spark-133284bb-7c9f-442d-8d5e-c5ea2dfd650e

agibsonccc · September 2, 2022, 9:36pm

@Nour-Rekik could you check the git hash you downloaded using git status? That nullpointer just being the resource seems very 1 off.
It fails here:

github.com

deeplearning4j/deeplearning4j/blob/18e6dac62e5e0219599ac445b93014ac95672e87/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/factory/Nd4j.java#L5148


      
          randomFactory = new RandomFactory(randomClazz);

          

          Class<? extends MemoryWorkspaceManager> workspaceManagerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(WORKSPACE_MANAGER));

          

          Class<? extends BlasWrapper> blasWrapperClazz = ND4JClassLoading.loadClassByName(pp.toString(BLAS_OPS));

          String clazzName = pp.toString(DISTRIBUTION, DefaultDistributionFactory.class.getName());

          Class<? extends DistributionFactory> distributionFactoryClazz = ND4JClassLoading.loadClassByName(clazzName);

          

          

          memoryManager = memoryManagerClazz.newInstance();

          constantHandler = constantProviderClazz.newInstance();

          if(shapeInfoProviderClazz != null)

              shapeInfoProvider = shapeInfoProviderClazz.newInstance();

          if(workspaceManagerClazz != null)

              workspaceManager = workspaceManagerClazz.newInstance();

          

          Class<? extends OpExecutioner> opExecutionerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(OP_EXECUTIONER, DefaultOpExecutioner.class.getName()));

We obtain the reference to that here:

github.com

deeplearning4j/deeplearning4j/blob/18e6dac62e5e0219599ac445b93014ac95672e87/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/factory/Nd4j.java#L5132


      
                  .loadClassByName(pp.toString(CONVOLUTION_OPS, DefaultConvolutionInstance.class.getName()));

          String defaultName = pp.toString(DATA_BUFFER_OPS, "org.nd4j.linalg.cpu.nativecpu.buffer.DefaultDataBufferFactory");

          Class<? extends DataBufferFactory> dataBufferFactoryClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(DATA_BUFFER_OPS, defaultName));

          Class<? extends BaseShapeInfoProvider> shapeInfoProviderClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(SHAPEINFO_PROVIDER));

          

          Class<? extends BasicConstantHandler> constantProviderClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(CONSTANT_PROVIDER));

          

          Class<? extends BasicMemoryManager> memoryManagerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(MEMORY_MANAGER));

          

          allowsOrder = backend.allowsOrder();

          String rand = pp.toString(RANDOM_PROVIDER, DefaultRandom.class.getName());

          Class<? extends org.nd4j.linalg.api.rng.Random> randomClazz = ND4JClassLoading.loadClassByName(rand);

          randomFactory = new RandomFactory(randomClazz);

          

          Class<? extends MemoryWorkspaceManager> workspaceManagerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(WORKSPACE_MANAGER));

What’s odd is some of the references work which would imply that this specific reference is missing.

I’ll try to see if I can reproduce this locally. I just want to make sure this is a recent version and I"m not chasing something that’s just an old dl4j version or something.

Nour-Rekik · September 2, 2022, 10:02pm

Okay thank you!
Here the git status of my project directory:

nore667e@taurusi8030:~/bigdl_pipeline> git status
# On branch master
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    img.png
#       modified:   pom.xml
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/data/StreamClass.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/imageNet_image/DataProcessing.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/imageNet_image/streaming/NetworkRetraining.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/imageNet_image/streaming/NetworkRetrainingMain.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/multi_layer_perceptron_mnist/MnistNetwork.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/multi_layer_perceptron_mnist/NeuralNetworkConfig.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/streaming_approach/MnistNetworkRetraining.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/streaming_approach/MnistRetrainingMain.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/traditional_batch/MnistNetworkBatchMain.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/mnist_image/traditional_batch/MnistNetworkBatchProcessing.java
#       modified:   src/main/java/com/examples/DeepLearningOnSpark/start/SparkContext.java
#       deleted:    src/main/java/com/examples/DeepLearningOnSpark/tiny_imageNet/CnnNetworkFactory.java
#       deleted:    src/main/java/com/examples/DeepLearningOnSpark/tiny_imageNet/PreprocessSpark.java
#       deleted:    src/main/java/com/examples/DeepLearningOnSpark/tiny_imageNet/TrainSpark.java
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       MnistMLPMain.java
#       deepLearningSimpleOne.iml
#       env.out
#       evaluation.csv
#       src/main/java/com/examples/DeepLearningOnSpark/imageNet_image/batch_processing/
#       src/main/resources/
no changes added to commit (use "git add" and/or "git commit -a")

and here the git status of the deeplearning4j directory:

nore667e@taurusi8030:~/deeplearning4j> git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   libnd4j/pom.xml
#       modified:   nd4j/nd4j-backends/nd4j-backend-impls/nd4j-cuda/src/main/java/org/nd4j/linalg/jcublas/bindings/Nd4jCuda.java
#
#
# It took 3.68 seconds to enumerate untracked files. 'status -uno'
# may speed it up, but you have to be careful not to forget to add
# new files yourself (see 'git help status').
no changes added to commit (use "git add" and/or "git commit -a")

agibsonccc · September 2, 2022, 10:15pm

@Nour-Rekik Sorry could you try this?

git rev-parse HEAD

I don’t see the hash in there. I assume this clone is from the last few days? If so that should be enough for me to look at this.

Nour-Rekik · September 2, 2022, 10:28pm

Yes sure here the output:

nore667e@taurusi8030:~/deeplearning4j> git rev-parse HEAD
b004b31feba6852f5dd2ce391d045dc480ac885c

agibsonccc · September 2, 2022, 10:36pm

Ok based on that I see what commit you’re on. I can work with this. Let me take a look locally. Thanks!

Nour-Rekik · September 2, 2022, 10:39pm

Okay thank you so much!

Topic		Replies	Views
Can't run DL4J with GPU on Linux DL4J	7	460	July 22, 2021
DL4J based program reported error about native openblas support DL4J	5	1301	March 8, 2021
UnsatisfiedLinkError with cudnn on Windows 10 DL4J	8	1387	September 18, 2020
SameDiff example MNISTCNN fails for Cuda, jnicudart.dll SameDiff	2	752	October 20, 2021
Requesting help with POM for GPU support GeForce RTX 3090 DL4J	16	1093	July 17, 2021

Version `GLIBC_2.23' not found when using dl4j-cuda

Related Topics