No such method error when init model on snapshot

MPdaedalus · May 18, 2021, 3:58pm

16:55:28.501 [main] ERROR org.deeplearning4j.common.config.DL4JClassLoading - Cannot create instance of class ‘org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper’.
java.lang.NoSuchMethodException: org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.(java.lang.Class,org.nd4j.linalg.api.buffer.DataType)
at java.base/java.lang.Class.getConstructor0(Class.java:3517)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2691)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:103)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:89)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:74)
at org.deeplearning4j.nn.layers.HelperUtils.createHelper(HelperUtils.java:58)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.initializeHelper(ConvolutionLayer.java:75)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.(ConvolutionLayer.java:70)
at org.deeplearning4j.nn.conf.layers.ConvolutionLayer.instantiate(ConvolutionLayer.java:173)
at org.deeplearning4j.nn.conf.graph.LayerVertex.instantiate(LayerVertex.java:106)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:572)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:437)

my pom.xml

<dependencies>
<dependency>
		<groupId>org.eclipse.collections</groupId>
		<artifactId>eclipse-collections-api</artifactId>
		<version>10.4.0</version>
	</dependency>
	<dependency>
		<groupId>org.eclipse.collections</groupId>
		<artifactId>eclipse-collections</artifactId>
		<version>10.4.0</version>
	</dependency>
<dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-ui</artifactId>
        <version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.nd4j</groupId>
		<artifactId>nd4j-cuda-11.2-preset</artifactId>
		<version>${dl4j.version}</version>
		<classifier>linux-x86_64-cudnn</classifier>
	</dependency>
	<dependency>
		<groupId>org.deeplearning4j</groupId>
		<artifactId>deeplearning4j-core</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.deeplearning4j</groupId>
		<artifactId>deeplearning4j-cuda-11.2</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.deeplearning4j</groupId>
		<artifactId>deeplearning4j-zoo</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.deeplearning4j</groupId>
		<artifactId>deeplearning4j-datavec-iterators</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.datavec</groupId>
		<artifactId>datavec-local</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
</dependencies>

I have cudnn installed and cuda 11.2

i’ve checked my class path and .m2/repository/org/deeplearning4j/deeplearning4j-cuda-11.2/1.0.0-SNAPSHOT/deeplearning4j-cuda-11.2-1.0.0-SNAPSHOT.jar is on it and checked the jar and the CudnnConvolutionHelper.class (32.1KB) is there. I tried to debug it and the class loader does actually find the class in the jar but something goes wrong when it is instantiated to an actual class object, something to do with the constructor it seems.

I can also confirm that org.nd4j.linalg.api.buffer.DataType.class is present in .m2/repository/org/nd4j/nd4j-api/1.0.0-SNAPSHOT/nd4j-api-1.0.0-SNAPSHOT.jar

tested with jdk 8 and 16, same result

treo · May 19, 2021, 3:16pm

This looks like something we’ve fixed already but that may not have propagated to the snapshots yet. If it still doesn’t work with the snapshots tomorrow, ping us again please.

MPdaedalus · May 19, 2021, 3:31pm

ok, have same issue on LeNetMNISTReLu example so i’ll wait
currently using deeplearning4j-cuda-11.2-1.0.0-20210514.061745-66, nothing else up yet

agibsonccc · May 21, 2021, 3:36am

@MPdaedalus Just deeplearning4j-cuda-11.2 won’t require you to compile any c++. Just clone the project, cd in to the root directory and run:

./change-cuda-versions.sh 11.2
cd deeplearning4j/deeplearning4j-nn && mvn clean install -DskipTests 
cd ../deeplearning4j-cuda-11.2 && mvn clean install -DskipTests

Beyond that, which OS are you on? I’ll try to make sure snapshots get pushed for you.
Thanks!

MPdaedalus · May 22, 2021, 10:24pm

on fedora linux 32, i’ve switched back to beta7 for now until the snapshots are a bit more stable as it does not have this issue.

agibsonccc · May 22, 2021, 10:55pm

@MPdaedalus thanks. Yeah that’s fine. My suggestion did workfor another user and should keep your setup running, but I understand the hesitation in wanting to deal with cloning the code base.

The main issue atm is I’m simultaneously doing release testing + the need to do manual snapshots atm. Ideally this whole process would be automated, but with github actions quotas, they kill your jobs if you run too many at once. This is on top of sonatype purging snapshots every 3 days or so. I’ll work out a more sustainable solution after the release is done.
It should be this week from the looks of things. I’ll let everyone know after final release testing is done.

MPdaedalus · May 22, 2021, 10:58pm

sounds good. thanks again

agibsonccc · May 27, 2021, 4:28am

@MPdaedalus JFYI, 1.0.0-M1 binaries are available on maven central. Docs are still WIP yet, but please feel free to ask questions.

MPdaedalus · May 27, 2021, 7:29pm

thanks for the info, will check them out on monday, Thanks for all the great work you’ve put in, I don’t know where i’d be without dl4j as i’ve only been a java dev for two decades and refuse to use dynamically typed languages like python due to debug nightmare/performance problems (been there two decades ago and not going back) so other than a half assed java api to something else like tensorflow/mxnet, DL4J is my only option.

agibsonccc · May 27, 2021, 10:32pm

@MPdaedalus sure! There’s still a lot of work to do though! If there’s anything missing from those frameworks that you need, please do keep filing feature requests.

MPdaedalus · June 1, 2021, 11:46am

just tested M1 but I still get the same error although this method also returns NoSuchMethod error when also adding listeners or loading training examples before running model but they are handled successfully. The error does not crop up in beta7 with same code.

The error occurs because arrayContentsEq returns false as the two params don’t match

where
constructor=public org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper(org.nd4j.linalg.api.buffer.DataType)
parameterTypes=[class java.lang.Class, class [Ljava.lang.Object;]
and fact.getExecutableSharedParameterTypes(constructor) returns>
[class org.nd4j.linalg.api.buffer.DataType]

at Class.class line 3506
with code>
private Constructor getConstructor0(Class<?> parameterTypes,
int which) throws NoSuchMethodException
{
ReflectionFactory fact = getReflectionFactory();
Constructor constructors = privateGetDeclaredConstructors((which == Member.PUBLIC));
for (Constructor constructor : constructors) {
if (arrayContentsEq(parameterTypes,
fact.getExecutableSharedParameterTypes(constructor))) {
return constructor;
}
}
throw new NoSuchMethodException(methodToString(“”, parameterTypes));
}

java.lang.NoSuchMethodException: org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.(java.lang.Class,[Ljava.lang.Object;)
at java.base/java.lang.Class.getConstructor0(Class.java:3517)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2691)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:103)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:89)
at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:74)
at org.deeplearning4j.nn.layers.HelperUtils.createHelper(HelperUtils.java:57)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.initializeHelper(ConvolutionLayer.java:72)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.(ConvolutionLayer.java:67)
at org.deeplearning4j.nn.conf.layers.ConvolutionLayer.instantiate(ConvolutionLayer.java:173)
at org.deeplearning4j.nn.conf.graph.LayerVertex.instantiate(LayerVertex.java:106)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:572)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:437)

this leads to runtimeexception at DL4JClassLoading.class line 108

agibsonccc · June 1, 2021, 11:59am

@MPdaedalus DM me a test case that I can just run and make sure we integrate in to our test suite. That way we can turn around a quick fix.

MPdaedalus · June 1, 2021, 12:06pm

just to double check to make sure something stupid in my config is not wrong, is it normal to get these can not find class messages in the first few lines?

OpenJDK 64-Bit Server VM warning: Option --illegal-access is deprecated and will be removed in a future release.
12:26:19.695 [main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
12:26:19.712 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.nd4j.linalg.jblas.JblasBackend] of provided class-loader.
12:26:19.713 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.canova.api.io.data.DoubleWritable] of provided class-loader.
12:26:19.716 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.nd4j.linalg.jblas.JblasBackend] of provided class-loader.
12:26:19.717 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.canova.api.io.data.DoubleWritable] of provided class-loader.
12:26:22.197 [main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 32
12:26:22.243 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Linux]
12:26:22.243 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [4]; Memory: [3.9GB];
12:26:22.243 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [CUBLAS]
12:26:22.260 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - ND4J CUDA build version: 11.2.152
12:26:22.263 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - CUDA device 0: [NVIDIA GeForce GTX 1070]; cc: [6.1]; Total memory: [8513585152]
12:26:22.263 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - Backend build information:
GCC: “7.5.0”
STD version: 201402L
CUDA: 11.2.152
DEFAULT_ENGINE: samediff::ENGINE_CUDA
HAVE_FLATBUFFERS

agibsonccc · June 1, 2021, 12:08pm

@MPdaedalus those are version checks to make sure older versions of the library aren’t present on the classpath See: Move version check warnings to tracing · Issue #9299 · eclipse/deeplearning4j · GitHub

agibsonccc · June 1, 2021, 12:09pm

Highlighting 1.0.0-M1 bug DL4JClassLoading of CuDNN · Issue #9331 · eclipse/deeplearning4j · GitHub for you as well @jjijiji

MPdaedalus · June 1, 2021, 12:13pm

I think jljljl just nailed it, its the exact same problem with the wrong methods overloading.

is jljljl commit integrated into the latest snapshot yet because I would like to test it to see if it fixes the problem.

agibsonccc · June 1, 2021, 12:15pm

@MPdaedalus not yet, but you’re welcome to pull his branch and try it. I wanted to make sure we have test cases in his PR to prevent regressions.

MPdaedalus · June 1, 2021, 12:17pm

ok, i’ll give his branch a try.

agibsonccc · June 1, 2021, 12:21pm

@MPdaedalus keep in mind that you just need to install the deeplearning4j-nn submodule, not the whole thing. The core issue is in HelperUtils. Everything else just uses that. That way you avoid chasing rabbit holes for now if you want to use cudnn, consider giving the new nd4j classifiers a try. You can find WIP docs here: GitHub - KonduitAI/deeplearning4j-gitbook-sync at en-1.0.0-M1 Relevant doc here: deeplearning4j-gitbook-sync/config-cudnn.md at en-1.0.0-M1 · KonduitAI/deeplearning4j-gitbook-sync · GitHub

We’ll promote the new docs after we’re finished with the redirects. Otherwise google will get cranky.

MPdaedalus · June 1, 2021, 12:33pm

yea I spotted them when browsing sonatype.org so started using them for the snapshots. I think classifiers are the way forward to stop everything being pulled in needlessly. Watching stuff like ffmpeg-macosx getting pulled with cuda-platform-redist is kinda odd lol.

Using deeplearning4j-nn and not having to build everything on my crappy cpu is a relief.

Topic		Replies	Views
java.lang.ClassNotFoundException: org.nd4j.nativeblas.OpaqueDataBuffer ND4J	4	1595	June 13, 2020
LSTMHelper error with GPU backend DL4J	2	567	May 23, 2021
Failed to build dl4j-examples, compilation error DL4J	2	295	September 9, 2022
DenseLayer (index=9, name=ffn0) nIn=0, nOut=3072; nIn and nOut must be > 0 RL4J	18	587	August 1, 2023
Error in Eclipse: Provider class com.twelvemonkeys.imageio.plugins.jpeg.JPEGImageReaderSpi not in module DataVec	3	1031	November 2, 2020

No such method error when init model on snapshot

Related topics