Nd4j-cuda-11.2-platform not available for m2

ND4J
1

Good day @agibsonccc ,
I am try to use dl4j m2. This is the error am getting:

image
image1677×1035 75.2 KB

Here is my sample pom.xml file

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>training</groupId>
  <artifactId>training</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <release>17</release>
        </configuration>
      </plugin>
      <plugin>
  <groupId>kr.motd.maven</groupId>
  <artifactId>os-maven-plugin</artifactId>
  <version>1.7.0</version>
  <executions>
    <execution>
      <phase>initialize</phase>
      <goals>
        <goal>detect</goal>
      </goals>
    </execution>
  </executions>
</plugin>

    </plugins>
    
  </build>
  <properties>
  		<nd4j.cpu.backend>nd4j-native-platform</nd4j.cpu.backend>
  		<nd4j.cpu.backend2>nd4j-native</nd4j.cpu.backend2>
        <nd4j.gpu.backend>nd4j-cuda-11.2-platform</nd4j.gpu.backend> 
        <!-- <cuda.redist.10.1.version>10.1-7.6-1.5.2</cuda.redist.10.1.version> -->
        <cuda.redist.11.2.version>11.2-8.1-1.5.5</cuda.redist.11.2.version>
        <dl4j.version>1.0.0-M2</dl4j.version>
        <ffmpeg.version>4.1-1.4.4</ffmpeg.version>
        <javacv.version>1.5.5</javacv.version>
        <logback.version>1.1.7</logback.version> 
        <jackson.version>2.10.2</jackson.version>
    </properties>
    <repositories>
     <repository>
            <id>sonatype-nexus-snapshots</id>
            <name>Sonatype Nexus Snapshots</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>
        <repository>
            <id>M2</id>
            <url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1019</url>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
<repository>
            <id>M2-1</id>
            <url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1032</url>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
   <repository>
            <id>M2-2</id>
            <url>https://s01.oss.sonatype.org/content/repositories/orgdeeplearning4j-1034</url>
            <releases>
                <enabled>true</enabled>
            </releases>
        </repository>
    </repositories>
    
    
    <dependencies>
  	<dependency>
         <groupId>org.bytedeco</groupId>
         <artifactId>opencv-platform</artifactId>
         <version>4.5.1-1.5.5</version>
     </dependency>
       <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.cpu.backend}</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.gpu.backend}</artifactId>
            <version>${dl4j.version}</version>
        </dependency>  
        
         <!-- https://mvnrepository.com/artifact/org.bytedeco/cuda-platform-redist -->
         <!-- 
		<dependency>
		    <groupId>org.bytedeco</groupId>
		    <artifactId>cuda-platform-redist</artifactId>
		    <version>${cuda.redist.10.2.version}</version>
		</dependency>    
        -->
               
        <!-- https://mvnrepository.com/artifact/org.nd4j/nd4j-common -->
		<dependency>
		    <groupId>org.nd4j</groupId>
		    <artifactId>nd4j-common</artifactId>
		    <version>${dl4j.version}</version>
		</dependency>
        
        <dependency>
            <groupId>org.datavec</groupId>
            <artifactId>datavec-api</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.datavec</groupId>
            <artifactId>datavec-data-image</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.datavec</groupId>
            <artifactId>datavec-local</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-datasets</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-zoo</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <!-- ParallelWrapper & ParallelInference live here -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-parallel-wrapper</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
    
    	<!-- Log dependency -->
		<dependency>
		    <groupId>org.slf4j</groupId>
		    <artifactId>slf4j-api</artifactId>
		    <version>1.7.25</version>
		</dependency>
                                                       
        <dependency>                                                            
            <groupId>ch.qos.logback</groupId>                                   
            <artifactId>logback-classic</artifactId>                            
            <version>${logback.version}</version>                               
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-utility-iterators</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-modelimport</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        
        <!-- https://mvnrepository.com/artifact/org.bytedeco/openblas-platform -->
        <!--  
		<dependency>
		    <groupId>org.bytedeco</groupId>
		    <artifactId>openblas-platform</artifactId>
		    <version>0.3.9-1.5.3</version>
		</dependency>
		-->
        <dependency>
           <groupId>org.bytedeco.javacpp-presets</groupId>
           <artifactId>ffmpeg</artifactId>
           <version>${ffmpeg.version}</version>
   	    </dependency>
   	    <dependency>
		    <groupId>org.bytedeco</groupId>
		    <artifactId>javacpp</artifactId>
		    <version>1.5.5</version>
		  </dependency>
   	    <!-- Jackson dependencies -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
          <groupId>com.fasterxml.jackson.core</groupId>
		  <artifactId>jackson-databind</artifactId>
		  <version>${jackson.version}</version>
        </dependency>
                
          <!-- 
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-cuda-10.1</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
		    <groupId>org.deeplearning4j</groupId>
		    <artifactId>deeplearning4j-cuda-10.1</artifactId>
		    <version>${dl4j.version}</version>
		</dependency> 
		    -->
     
  </dependencies>
</project>

Is M2 missing ND4J-cuda-11.2-platoform?

2

@ajmakoni we didn’t publish cuda 11.2 in general. It’s been difficult to maintain 2. You can use cuda 11.4 now.

3

Is -platform published for the M2 release?

Looking on Maven Central, I found nd4j-cuda-11.4-preset but not nd4j-cuda-11.4-platform. Was “platform” renamed to “preset” in the M2 release, or does “preset” mean something else?

Thanks for helping clarify.

5

@ccjernigan ah it looks like the platform module never got published. Let me get to that next week.
For now just uses the -platform pom (which is just a bill of materials anyways).

Sorry the cuda builds on github actions are a bit inconsistent so sometimes manual intervention is needed and details are missed. Better automation is a goal but in my experience is hard to do with the sonatype releases.

6

Are you still planning to resolve this issue? I don’t see the platform artifact published yet on Maven Central and this is a blocker for me.

7

@ccjernigan sorry never got around to this. Thanks for the ping. I ended up prioritizing internal customers/users and the next release.

The specific problem around the cuda builds was the slow infrastructure and error prone waiting that happens with github actions.

I know you as an end user just care whether it’s there or not but the underlying infra there forces me to make a trade off on my own time as well as just prioritizing problems like this.

A lot of this came down to cuda’s compilation time causing github actions to time out. This is much better with the cuda 11.6 release and later. We also ended up moving on from cuda 11.2.

What I’ll try to do at least is see if we can get the module published. I made the cuda build more flexible to allow for different cuda versions to be specified which will allow me to at least build cuda 11.2 from the build infra. Note that from next release we’ll be only publishing cuda 11.4 and 11.6

In the mean time, -platform modules aren’t mandatory just a quality of life improvement. You can just specify cuda-11.2 classifiers individually as well. In that case you just need to setup specifying cuda and other native dependencies as declared in the -platform pom here:

If you’d like I can at least help you with the workaround while I see how the cuda build goes.

8

@ccjernigan you can find nd4j-cuda-11.2-platform in maven central now. I managed to setup a more robust github actions profile which worked to build the right version and deploy it. Thanks for the ping. Hopefully that addresses the issues.
Let me know if there’s anything else. Please also do follow up with me if something drops.

9 
<properties>
	<nd4j.backend>nd4j-cuda-11.6</nd4j.backend>
	<java.version>17</java.version>
	<dl4j.version>1.0.0-M2</dl4j.version>
</properties>
<dependencies>
	<dependency>
		<groupId>org.nd4j</groupId>
		<artifactId>${nd4j.backend}</artifactId>
		<version>${dl4j.version}</version>
	</dependency>
	<dependency>
		<groupId>org.nd4j</groupId>
		<artifactId>${nd4j.backend}</artifactId>
		<version>${dl4j.version}</version>
		<classifier>linux-x86_64</classifier>
	</dependency>

When compile for above setting in Windows, maven will try to download the jars of windows version also, but there are no the jars of windows version in repo1.maven.org/maven2, how to do to only try to download the jars of linux version?

10

@SidneyLann -Djavacpp.platform=linux-x86_64 in your maven build. Same as always.

11

Yes, it work. Thanks.