Hi,
So I had always presumed that aarch (64bit) systems were the same as arm (64bit) systems.
But when deploying my Minecraft server plugin on arm it works perfectly, but for aarch I get a NoAvailableBackendException.
How can I solve this?
Can you provide some additional information? What chip is it? What OS are you using? (In particular which version of that OS).
It may be that it is some missing library or an old library version that is the problem.
Here is the system info:
System Info: Java 17 (OpenJDK 64-Bit Server VM 17.0.2+8-86) Host: Linux 5.15.0-1016-oracle (aarch64)
I am deploying my DL4j libraries using an uberjar with -Djavacpp.platform=arm64
Needless to say, I am using nd4j-native-platform and deeplearning4j-core
Thanks!
@Joehot200 nd4j-native-platform was mainly meant for use with intel systems…you’ll just need to do what nd4j-native-platform does for intel platforms manually.
That means the following:
Use nd4j-native without a classifier (this includes the java classes)
Declare an additional dependency on nd4j-native with a classifier (in your case linux-arm64 is what you want)
Then do the same for openblas. (openblas without a classifier and openblas with a linux-arm64 classifier)
I feel like I’ve tried throwing everything in at this point, but it still does not work. Here is my POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>joehot200</groupId>
<artifactId>AntiAuraML-windows-64</artifactId>
<version>2.0</version>
<properties>
<dl4j.version>1.0.0-M1</dl4j.version>
<fully.qualified.main.class>main.java.KillauraML.KillauraML</fully.qualified.main.class>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>1.0.0-M1</version>
<!--<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>-->
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-nn</artifactId>
<version>${dl4j.version}</version>
<!--<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>-->
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-modelimport</artifactId>
<version>${dl4j.version}</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native-platform</artifactId>
<version>1.0.0-M2</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>1.0.0-M2</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>1.0.0-M2</version>
<classifier>linux-arm64</classifier>
<!-- <classifier>windows-x86_64</classifier> -->
</dependency>
<!-- https://mvnrepository.com/artifact/org.bytedeco/openblas -->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>openblas</artifactId>
<version>0.3.21-1.5.8</version>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>openblas</artifactId>
<version>0.3.21-1.5.8</version>
<classifier>linux-arm64</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>1.5.8</version>
<classifier>linux-arm64</classifier>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.maven:lib:tests</exclude>
<exclude>log4j:log4j:jar:</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
And here is the error:
[15:43:25 WARN]: java.lang.ExceptionInInitializerError
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.<init>(NativeOpExecutioner.java:79)
[15:43:25 WARN]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[15:43:25 WARN]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[15:43:25 WARN]: at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[15:43:25 WARN]: at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[15:43:25 WARN]: at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[15:43:25 WARN]: at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
[15:43:25 WARN]: at java.base/java.lang.Class.newInstance(Class.java:645)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5178)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5093)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:270)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.deeplearning4j.util.ModelSerializer.restoreMultiLayerNetworkHelper(ModelSerializer.java:282)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.deeplearning4j.util.ModelSerializer.restoreMultiLayerNetwork(ModelSerializer.java:237)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.deeplearning4j.util.ModelSerializer.restoreMultiLayerNetwork(ModelSerializer.java:221)
[15:43:25 WARN]: at AntiAura_V2.jar//joehot200.ah.b(ah.java:142)
[15:43:25 WARN]: at AntiAura_V2.jar//joehot200.bi.b(bi.java:106)
[15:43:25 WARN]: at AntiAura_V2.jar//joehot200.bk.run(bk.java:296)
[15:43:25 WARN]: at org.bukkit.craftbukkit.v1_18_R2.scheduler.CraftTask.run(CraftTask.java:101)
[15:43:25 WARN]: at org.bukkit.craftbukkit.v1_18_R2.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:483)
[15:43:25 WARN]: at net.minecraft.server.MinecraftServer.b(MinecraftServer.java:1532)
[15:43:25 WARN]: at net.minecraft.server.dedicated.DedicatedServer.b(DedicatedServer.java:483)
[15:43:25 WARN]: at net.minecraft.server.MinecraftServer.a(MinecraftServer.java:1456)
[15:43:25 WARN]: at net.minecraft.server.MinecraftServer.w(MinecraftServer.java:1226)
[15:43:25 WARN]: at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:316)
[15:43:25 WARN]: at java.base/java.lang.Thread.run(Thread.java:833)
[15:43:25 WARN]: Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: https://deeplearning4j.konduit.ai/nd4j/backend
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:116)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:37)
[15:43:25 WARN]: ... 25 more
[15:43:25 WARN]: Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
[15:43:25 WARN]: at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)
[15:43:25 WARN]: at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
[15:43:25 WARN]: at java.base/java.lang.System.loadLibrary(System.java:1989)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1718)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.bytedeco.javacpp.Loader.load(Loader.java:1328)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.bytedeco.javacpp.Loader.load(Loader.java:1132)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.linalg.cpu.nativecpu.bindings.Nd4jCpu.<clinit>(Nd4jCpu.java:14)
[15:43:25 WARN]: at java.base/java.lang.Class.forName0(Native Method)
[15:43:25 WARN]: at java.base/java.lang.Class.forName(Class.java:467)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:88)
[15:43:25 WARN]: ... 26 more
[15:43:25 WARN]: Caused by: java.lang.UnsatisfiedLinkError: /home/container/?/.javacpp/cache/AntiAuraML-all64.jar/org/nd4j/linalg/cpu/nativecpu/bindings/linux-arm64/libjnind4jcpu.so: libgomp.so.1: cannot open shared object file: No such file or directory
[15:43:25 WARN]: at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
[15:43:25 WARN]: at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
[15:43:25 WARN]: at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
[15:43:25 WARN]: at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
[15:43:25 WARN]: at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
[15:43:25 WARN]: at java.base/java.lang.Runtime.load0(Runtime.java:755)
[15:43:25 WARN]: at java.base/java.lang.System.load(System.java:1953)
[15:43:25 WARN]: at AntiAuraML-all64.jar//org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1668)
[15:43:25 WARN]: ... 34 more
Note that while I am using 1.0.0-M1 here, I also tried the same with 1.0.0-M2.
Thanks
Joe
As I guessed. It is because it can’t find an openmp lib.
As you can see here, we ship that with the x86_64 version, but we don’t do that with the arm version.
I guess, no one else complained about that until now, because that is a lib that is often installed by default.
On an Ubuntu based system you can install it with apt install libgomp1 .
How would I make this work in a situation where a person doesn’t have access to the machine? (aka presume you can’t apt-get or install anything)
I tried manually putting the libgomp.so.1 file in the arm64 folder, but that did nothing.
@Joehot200 this is for a minecraft plugin right? There are different ways you could manually configure the path.
If you can help me understand a bit I can make a good recommendation. Usually there are different ways to manually set the loader path to discover libraries. You could bundle the appropriate libgomp using that. Could you clarify what you expect users to be able to do? Also, what ubuntu system are they running on? Is it the default pi distro?
Minecraft has this concept of “shared hosts”. Basically, the host has a web panel which will run the server jar (spigot.jar). Spigot jar then loads my plugin (let’s say plugin.jar) onto the classpath.
In the shared host, the user only has access to the folder with “spigot.jar” (+ subfolders, including plugins folder). Here are a couple screenshots:
It is difficult to predict which exact OS a shared host uses. Ideally I would like the plugin to be as accessible as possible. This particular shared host uses aarm64.
To make DL4j work for plugin.jar, you already helped me a few months ago with this line of code:
DL4JClassLoading.setDl4jClassloader(this.getClassLoader());
@Joehot200 as a workaround could I help you in setting up a way for you to bundle/load the .so files in the jars then?
We can also figure out all the various .so files you need and maybe include some of them. I’ll want to scrutinize what we include by default since many arm64 systems are embedded though.
Absolutely, you could! I can load anything that’s in the plugin.jar files, and pretty much do whatever I want with files programmatically.
Even if you don’t include them by default it would be great to know what I need to bundle in with it manually.
@Joehot200 generally that will involve setting LD_LIBRARY_PATH or something similar on startup. That or some custom way of setting up the jars. What can you configure when it starts? JVM properties? Environment variables?
Sorry for the late reply, Merry Christmas! The vast majority of shared hosts allow the modification of startup flags, so I would be happy with a solution that involved them.
Once the plugin has been initiated you have pretty much all power programatically, same as the server jar does (e.g. a malicious plugin could theoretically wipe your entire server).
Any update on this? Would be really helpful if you have a solution for me!
@Joehot200 sorry holidays. Unfortunately, lunar new year (mid jan) factors in for me as well.
I’m back at my desk and just saw this thanks for the ping.
Looking at this again, one other question I’d have would be on the glibc version.
I’m going to do a PR for this: Preload libgomp on arm architectures by agibsonccc · Pull Request #9897 · deeplearning4j/deeplearning4j · GitHub
Apologies, I’ve been busy myself! I see the PR has been rolled into DL4j, does this mean I simply need to update?
How would I check the glibc version?
Thanks!
You would need to build from source or wait for release. I am on vacation right now but will be finishing that next week. Just need to finalize cuda. Cpu is ready.