ND4J compile from source fails with snapshot

ND4J
1

Trying to compile DL4J from scratch as I need to modify some core classes.

Following https://deeplearning4j.konduit.ai/getting-started/build-from-source

however when building ND4j after libnd4j was successfully built I get the following error when building for gpu using>
mvn clean install -DskipTests -Dmaven.javadoc.skip=true -pl ‘!:nd4j-tests’
on latest SNAPSHOT

[ERROR] Failed to execute goal com.lewisd:lint-maven-plugin:0.0.11:check (pom-lint) on project nd4j: Execution pom-lint of goal com.lewisd:lint-maven-plugin:0.0.11:check failed: An API incompatibility was encountered while executing com.lewisd:lint-maven-plugin:0.0.11:check: java.lang.ExceptionInInitializerError: null
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>com.lewisd:lint-maven-plugin:0.0.11
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/daedalus/.m2/repository/com/lewisd/lint-maven-plugin/0.0.11/lint-maven-plugin-0.0.11.jar
[ERROR] urls[1] = file:/home/daedalus/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
[ERROR] urls[2] = file:/home/daedalus/.m2/repository/org/sonatype/sisu/sisu-inject-bean/1.4.2/sisu-inject-bean-1.4.2.jar
[ERROR] urls[3] = file:/home/daedalus/.m2/repository/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar
[ERROR] urls[4] = file:/home/daedalus/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar
[ERROR] urls[5] = file:/home/daedalus/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
[ERROR] urls[6] = file:/home/daedalus/.m2/repository/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
[ERROR] urls[7] = file:/home/daedalus/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[8] = file:/home/daedalus/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] urls[9] = file:/home/daedalus/.m2/repository/org/springframework/spring-core/3.1.1.RELEASE/spring-core-3.1.1.RELEASE.jar
[ERROR] urls[10] = file:/home/daedalus/.m2/repository/org/springframework/spring-asm/3.1.1.RELEASE/spring-asm-3.1.1.RELEASE.jar
[ERROR] urls[11] = file:/home/daedalus/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
[ERROR] urls[12] = file:/home/daedalus/.m2/repository/org/springframework/spring-context/3.1.1.RELEASE/spring-context-3.1.1.RELEASE.jar
[ERROR] urls[13] = file:/home/daedalus/.m2/repository/org/springframework/spring-aop/3.1.1.RELEASE/spring-aop-3.1.1.RELEASE.jar
[ERROR] urls[14] = file:/home/daedalus/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
[ERROR] urls[15] = file:/home/daedalus/.m2/repository/org/springframework/spring-beans/3.1.1.RELEASE/spring-beans-3.1.1.RELEASE.jar
[ERROR] urls[16] = file:/home/daedalus/.m2/repository/org/springframework/spring-expression/3.1.1.RELEASE/spring-expression-3.1.1.RELEASE.jar
[ERROR] urls[17] = file:/home/daedalus/.m2/repository/org/jmock/jmock-junit4/2.5.1/jmock-junit4-2.5.1.jar
[ERROR] urls[18] = file:/home/daedalus/.m2/repository/org/jmock/jmock/2.5.1/jmock-2.5.1.jar
[ERROR] urls[19] = file:/home/daedalus/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar
[ERROR] urls[20] = file:/home/daedalus/.m2/repository/org/hamcrest/hamcrest-library/1.1/hamcrest-library-1.1.jar
[ERROR] urls[21] = file:/home/daedalus/.m2/repository/junit/junit-dep/4.4/junit-dep-4.4.jar
[ERROR] urls[22] = file:/home/daedalus/.m2/repository/com/thoughtworks/xstream/xstream/1.4.8/xstream-1.4.8.jar
[ERROR] urls[23] = file:/home/daedalus/.m2/repository/xmlpull/xmlpull/1.1.3.1/xmlpull-1.1.3.1.jar
[ERROR] urls[24] = file:/home/daedalus/.m2/repository/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar
[ERROR] urls[25] = file:/home/daedalus/.m2/repository/org/jdom/jdom/2.0.2/jdom-2.0.2.jar
[ERROR] urls[26] = file:/home/daedalus/.m2/repository/org/jdom/jaxen-jdom/1.0-FCS/jaxen-jdom-1.0-FCS.jar
[ERROR] urls[27] = file:/home/daedalus/.m2/repository/org/jdom/jaxen-core/1.0-FCS/jaxen-core-1.0-FCS.jar
[ERROR] urls[28] = file:/home/daedalus/.m2/repository/org/jdom/saxpath/1.0-FCS/saxpath-1.0-FCS.jar
[ERROR] urls[29] = file:/home/daedalus/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar
[ERROR] urls[30] = file:/home/daedalus/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar
[ERROR] urls[31] = file:/home/daedalus/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
[ERROR] urls[32] = file:/home/daedalus/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar
[ERROR] urls[33] = file:/home/daedalus/.m2/repository/org/apache/velocity/velocity-tools/2.0/velocity-tools-2.0.jar
[ERROR] urls[34] = file:/home/daedalus/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar
[ERROR] urls[35] = file:/home/daedalus/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar
[ERROR] urls[36] = file:/home/daedalus/.m2/repository/commons-chain/commons-chain/1.1/commons-chain-1.1.jar
[ERROR] urls[37] = file:/home/daedalus/.m2/repository/dom4j/dom4j/1.1/dom4j-1.1.jar
[ERROR] urls[38] = file:/home/daedalus/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar
[ERROR] urls[39] = file:/home/daedalus/.m2/repository/sslext/sslext/1.2-0/sslext-1.2-0.jar
[ERROR] urls[40] = file:/home/daedalus/.m2/repository/org/apache/struts/struts-core/1.3.8/struts-core-1.3.8.jar
[ERROR] urls[41] = file:/home/daedalus/.m2/repository/antlr/antlr/2.7.2/antlr-2.7.2.jar
[ERROR] urls[42] = file:/home/daedalus/.m2/repository/org/apache/struts/struts-taglib/1.3.8/struts-taglib-1.3.8.jar
[ERROR] urls[43] = file:/home/daedalus/.m2/repository/org/apache/struts/struts-tiles/1.3.8/struts-tiles-1.3.8.jar
[ERROR] urls[44] = file:/home/daedalus/.m2/repository/commons-jxpath/commons-jxpath/1.3/commons-jxpath-1.3.jar
[ERROR] urls[45] = file:/home/daedalus/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar
[ERROR] urls[46] = file:/home/daedalus/.m2/repository/commons-validator/commons-validator/1.4.1/commons-validator-1.4.1.jar
[ERROR] urls[47] = file:/home/daedalus/.m2/repository/org/sonatype/aether/aether-connector-wagon/1.13.1/aether-connector-wagon-1.13.1.jar
[ERROR] urls[48] = file:/home/daedalus/.m2/repository/org/apache/maven/wagon/wagon-http/2.9/wagon-http-2.9.jar
[ERROR] urls[49] = file:/home/daedalus/.m2/repository/org/apache/maven/wagon/wagon-http-shared/2.9/wagon-http-shared-2.9.jar
[ERROR] urls[50] = file:/home/daedalus/.m2/repository/org/jsoup/jsoup/1.7.2/jsoup-1.7.2.jar
[ERROR] urls[51] = file:/home/daedalus/.m2/repository/commons-io/commons-io/2.2/commons-io-2.2.jar
[ERROR] urls[52] = file:/home/daedalus/.m2/repository/org/apache/httpcomponents/httpclient/4.3.5/httpclient-4.3.5.jar
[ERROR] urls[53] = file:/home/daedalus/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar
[ERROR] urls[54] = file:/home/daedalus/.m2/repository/org/apache/httpcomponents/httpcore/4.3.2/httpcore-4.3.2.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[project>org.nd4j:nd4j:1.0.0-SNAPSHOT, parent: ClassRealm[maven.api, parent: null]]]
[ERROR] 
[ERROR] -----------------------------------------------------
[ERROR] : Unable to make field private final java.util.Comparator java.util.TreeMap.comparator accessible: module java.base does not "opens java.util" to unnamed module @5b332439

I have deleted my repository folder to insure everything is downloaded on demand so nothing is out of date but there seems to be some API incompatibility or java module problem? any ideas?

This is with fedora 32

2

That looks weird. What JDK are you using?

@agibsonccc have you seen that previously?

3

openjdk version “16” 2021-03-16
OpenJDK Runtime Environment 21.3 (build 16+36)
OpenJDK 64-Bit Server VM 21.3 (build 16+36, mixed mode, sharing)

I fixed the problem today by adding
export MAVEN_OPTS=“–illegal-access=warn”

I dunno if it is a maven issue or DL4J issue as I have built other projects with maven no problem. Its a problem that crops up when ever reflection is used.

see>reflection - How to solve InaccessibleObjectException ("Unable to make {member} accessible: module {A} does not 'opens {package}' to {B}") on Java 9? - Stack Overflow

I am still trying to build DL4j from scratch but running into various issues which take time to track down and fix. Here are some of the other issues that could be build bugs that I have worked around so far.

building nd4j>
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile (testCompile) on project nd4j-common: Compilation failure
[ERROR] Lombok visitor handler class lombok.javac.handlers.HandleVal failed: java.lang.NoSuchMethodError: ‘boolean com.sun.tools.javac.code.Symbol$TypeSymbol.isLocal()’

add to nd4j pom.xml

	<groupId>org.projectlombok</groupId>
	<artifactId>lombok</artifactId>
	<version>RELEASE</version>
	<scope>provided</scope>
</dependency>

maven complains about lombok but adding a dependency for it makes the error go away.

it is not made apparent in the build Docs that building libndj4 with blas is required to build nd4j even if you are using cuda. So using ./buildnativeoperations.sh -c cuda -cc 61 will cause a subsequent nd4j build failure, I am currently running ./buildnativeoperations.sh without cuda and then run it again with cuda (is this the right way to do this?)

I think there is a bug in ./buildnativeoperations.sh as it looks for openblas in the wrong directory (maybe on other distro like Ubuntu it is correct?)

On fedora openblas-devel is installed in /usr/include/openblas by default but the build scripts path is wrong>
line 548>
if [[ ! -f “$OPENBLAS_PATH/include/openblas_config.h” ]]; then
echo “Could not find OpenBLAS, please make sure to run the build with Maven or set the OPENBLAS_PATH variable”
OPENBLAS_PATH=“”
fi

So even if I set OPENBLAS_PATH enviroment variable correctly it will set OPENBLAS_PATH to “”;
I just copied the contents of /usr/include/openblas into usr/include and set my OPENBLAS_PATH=/usr
and it works, but this is only a workaround.

4

i’m a little confused, I can successfully build libnd4j and nd4j manually but when I then go onto build DL4J it seems to go onto build libnd4j and nd4j anyway which overrides the output of my previous manual builds and also removes my cuda build and does a cpu build from deeplearning4j/libnd4j/blasbuild.

So how do I build DL4J with cuda support if it builds libnd4j with cpu by default?

also the lombok bug also occurs in DL4J not just nd4j.

5

@MPdaedalus for now, build dl4j with java 8 just to avoid anything else. Please file an issue for jdk 16 and we’ll look in to this after the release.

As for the cpu build, you probably are missing some profiles.
Here’s how we build from source on linux:

mvn  -Possrh Djavacpp.platform=linux-x86_64   -Dlibnd4j.chip=cuda clean install -DskipTests

If you already have libnd4j compiled, ensure that the LIBND4J_HOME environment variable is defined in the right place.

6

I finally managed to build DL4J, I had to add -Dlibnd4j.compute=61 to your command above other wise the build would fail with nvcc trying to compile some 3000 series features and failing.

Only problem is that it always defaults to cuda 11 instead of cuda 11.2

I tried adding -Dlibnd4j.chip_version=11.2 but it always compiles with 11, do you know the correct switch to force 11.2?

I will get to work on the 1D Batch Normalization problem tomorrow.

7

For changing cuda versions, you should run the change-cuda-versions.sh script before building, so something like this:

# ./change-cuda-versions.sh 11.2

That should set the correct cuda and cudnn versions in every project.

1 Like