When I execute the Maven build, I do indeed use -Djavacpp.platform=android-arm parameter, yet the size of the uber jar is over 1 Gigabyte. I noticed one of the “INFO” tags that updates in Maven’s building progress, the following (among a multitude of other things) is something that pops up: [INFO] Including org.bytedeco:ffmpeg:jar:macosx-x86_64:5.0-1.5.7 in the shaded jar . Does this indicate that there are dependencies for MacOSX in my uber jar? If my understanding is correct from reading the link on reducing dependencies, then I would have guessed that this specific dependency (and who knows how many more) should not be included as part of my uber jar when I used the android-arm parameter.
Thanks.
LM
EDIT: Thanks for the sample program, I’ll check it out after seeing what can be done about my mammoth uber jar
Oh, I did not try “clean package,” just straight up “package.” I’ll try “clean package” then and see what that yields.
As far as running/executing the uber jar executable (not building/compiling to generate the jar files), I’ve indeed been using the java command line command of java -cp jar_name.jar.package_name.file_name. I didn’t know jars could be executed from Maven. That’ll be my next homework to learn about.
@LobsterMan if you use exec:java that’s actually not executing the jar. That’s just executing classes already compiled in your target directory. So if you use that approach make sure to run:
mvn clean compile
first.
Try to treat the jar build + java -cp and mvn exec:java as 2 different ways of accomplishing the same goal.
Quick question, is the parameter for specifying “android-arm” supposed to be:
-Djavcpp.platform=android-arm clean test
or
-Djavacpp.platform=android-arm clean test
Again, from the link Javacpp - Deeplearning4j it is specifically spelled out as -Djavcpp, without that “a”. I wasn’t sure if that would make a difference, or if it’s supposed to be missing the “a.”
My pleasure, glad I could be of help in some way. Indeed my Maven build/compile went down from ~30 minutes on my device to about 40 seconds when I included the “a.”
I’ll try the sample neural network program and I’ll reach out if I experience bumps along the way.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project dl4jTesting: Compilation failure
[ERROR] /storage/emulated/0/Download/dl4jTesting/src/main/java/crab/shrimp/program/App.java:[14,41] package org.deeplearning4j.examples.utils does not exist
Why is it not recognizing it (I’ve changed nothing in my pom file)? Although I intend to proceed with the mnist exercise, I will ultimately be trying to apply DL4J to numerical data for my desired application, using data most similar to the iris sample problem which is why this caught my attention.
@LobsterMan that comes from how you’re handling this. You’re not looking at what the example imports. That’s to be expected from randomly copying/pasting examples while not looking at its dependencies.
Also just try to understand that what’s there is just a normal every day programming in java error there.
ClassNotFound is a generic error specific to the programming language that you should also just understand exists and know how to solve within the context of any java project.
In this case, there’s a maven based java project and it can’t find a class. That will always throw a compilation error. Maven sets the classpath up for you when it runs/compiles everything. Make sure both the needed dependency declarations are there (again depends on the project) and the relevant classes in the project itself have the right package + class names.
Try to read the class before you copy and paste and make sure that you take the relevant dependencies and rename the appropriate packages.
Sorry that’s not easiest to use (it’s meant to be run in place) but there are many examples that have various utilities for loading data that make that easier. Make sure you pay more attention to that if you try to move something and you should be fine.
This makes sense, in hindsight, I didn’t look at the dependencies or alter my pom file to accomodate them. I’ve been too eager to get coding in Termux.
On the LeNetMNIST sample progrm, it built/compiled fine, but when I tried to run it after the successful compilatio , I received the following set of errors:
**Exception in thread “main” java.lang.ExceptionInInitializerError
at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.(NativeOpExecutioner.java:79)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
at java.base/java.lang.Class.newInstance(Class.java:558)
at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5129)
at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5044)
at org.nd4j.linalg.factory.Nd4j.(Nd4j.java:269)
at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:655)
at oscar.bastidas.program.App.main(App.java:55)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: https://deeplearning4j.konduit.ai/nd4j/backend
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:116)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:37)
… 11 more
Caused by: java.lang.UnsatisfiedLinkError: no jniopenblas_nolapack in java.library.path
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2541)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:873)
at java.base/java.lang.System.loadLibrary(System.java:1857)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1800)
at org.bytedeco.javacpp.Loader.load(Loader.java:1402)
at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
at org.bytedeco.openblas.global.openblas_nolapack.(openblas_nolapack.java:12)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.bytedeco.javacpp.Loader.load(Loader.java:1269)
at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
at org.nd4j.linalg.cpu.nativecpu.bindings.Nd4jCpu.(Nd4jCpu.java:14)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:88)
… 12 more
Caused by: java.lang.UnsatisfiedLinkError: Could not find jniopenblas_nolapack in class, module, and library paths.
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1767)
… 27 more**
I used the sample pom file we talked about previously to get me set up as-is (and it built fine). However, could it be there are other dependencies I needed to include to get the LeNetMNIST program actually running?
Exception in thread “main” java.lang.ExceptionInInitializerError
at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.(NativeOpExecutioner.java:79)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
at java.base/java.lang.Class.newInstance(Class.java:558)
at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5129)
at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5044)
at org.nd4j.linalg.factory.Nd4j.(Nd4j.java:269)
at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:655)
at oscar.bastidas.program.App.main(App.java:55)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: https://deeplearning4j.konduit.ai/nd4j/backend
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:116)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:37)
… 11 more
Caused by: java.lang.UnsatisfiedLinkError: no jniopenblas_nolapack in java.library.path
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2541)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:873)
at java.base/java.lang.System.loadLibrary(System.java:1857)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1800)
at org.bytedeco.javacpp.Loader.load(Loader.java:1402)
at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
at org.bytedeco.openblas.global.openblas_nolapack.(openblas_nolapack.java:12)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.bytedeco.javacpp.Loader.load(Loader.java:1269)
at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
at org.nd4j.linalg.cpu.nativecpu.bindings.Nd4jCpu.(Nd4jCpu.java:14)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:375)
at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:88)
… 12 more
Caused by: java.lang.UnsatisfiedLinkError: Could not find jniopenblas_nolapack in class, module, and library paths.
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1767)
… 27 more
<!--
Maven shade plugin configuration: this is required so that if you build a single JAR file (an "uber-jar")
it will contain all the required native libraries, and the backends will work correctly.
Used for example when running the following commants
mvn package
cd target
java -cp deeplearning4j-examples-1.0.0-beta-bin.jar org.deeplearning4j.LenetMnistExample
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>bin</shadedClassifierName>
<createDependencyReducedPom>true</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>org/datanucleus/**</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Hi, I was trying to get DL4J set up for Termux a while back but I was having some trouble getting it to run (it was compiling and building fine though). It seemed that I might have openblas not set up up properly. Would you please take a look at my pom file and see how I could get openblas set up correctly? Here’s my pom file in GitHub:
I have modified my pom file and it should have openblas in it (pom file can be seen at: startingDL4J/pom.xml at main · LobsterMan123/startingDL4J · GitHub). I am using the following maven command to compile/build my uber jar: mvn -Djavacpp.platform=android-arm clean package
but I still get an error when I run the uber jar with error saying “java.lang.UnsatisfiedLinkError: Could not find jniopenblas_nolapack in class, module, and library paths.”
Am I compiling/building with the wrong command? Are there additiobal dependencies? I’m lost as to why the project won’t run. Thanks.
@LobsterMan your openblas is WAY off. Please consult the exact version I gave you and just use that. Those are very incompatible.
Could you tell me where you got those dependencies from? I don’t think that is anywhere in our docs. Please do not get random artifacts off maven central. Guessing wastes both of our time.
I’ve been trying to get DL4J running on Termux since last year, 2022 so the dependencies come from the docs from that era since that’s the project I had set up. Regarding the openblas dependency in the link you gave me, it doesn’t take me to any version of openblas but to maven’s general search page, not any specific dependencies. I searched for ‘openblas’ in that link page and selected the lowest version of openblas since I recognize my project is a year old.
My apologies for not clarifying the age of my project earlier. Would you then please tell me what is the version of openblas I should be using? Thanks.
Note that this is for M2.1. Please feel free to ugprade.
Please get used to using maven central. It’ll have information for whatever build system you’re using.
Regarding your last part, M2 never depended on the old javacpp presets.
Stay away from anything with that namespace. The javacpp dependences migrated to org.bytedeco long ago. That’s the only thing you should be paying attention to.
Note that you can also get your dependencies running using nd4j-native-platform. Please read this for an overview: Required Dependencies - Deeplearning4j
Apologies for the complexity here but people have different use cases.
Usually the progression is as follows:
Start with -platform dependencies
They are surprised by large jar files due to downloading dependencies for all platforms.
We usually tell the person to do one of two things: use -Djavacpp.platform=yourplatform when building or manually declare dependencies.
When manually declaring dependencies then they need to pick their blas implementation. Usually this is openblas. You can reference the -platform pom here for examples: Maven Central Repository Search
Ok, I hope I’ve got my dependencies straight here (i.e. I got rid of the presets thing I had before and I’ve requested the most recent dependencies version). It turns out maven has some new way of showing requested dependencies which is why your original link automatically took me to a generic search page, but I got it resolved to see what you were directing me to (thank you for the links).
Despite these changes to the most recent versions, I’m still getting the error of it not being able to find jniopenblas_nolapack when I try to execute my uberjar. Another error in the mix also says nd4j is probably missing dependencies (?).
Trying to build with "mvn -DargLine=“org.bytedeco.javacpp.pathsfirst=true -Djavacpp.platform=android-arm” -Dorg.bytedeco.javacpp.pathsfirst=true -Djavacpp.platform=android-arm clean
yields a compile error saying: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Below is my complete dependencies section of my pom file (and I am indeed building/compiling as: mvn -Djavacpp.platform=android-arm clean package); the build works fine though, just not running the uberjar: