How to use the new generated DL4J artifact in DL4J-examples project?

agibsonccc · April 11, 2023, 7:11am

@yaoyao your console output doesn’t show that though. It says M2.1 right in the console output. Can you show me your full pom rather than what you think I need to see?

yaoyao · April 11, 2023, 7:14am

This is my pom in DL4J-EXAMPLES project:

<?xml version="1.0" encoding="UTF-8"?>

4.0.0

<groupId>org.deeplearning4j</groupId>
<artifactId>dl4j-examples</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>Introduction to DL4J</name>
<description>A set of examples introducing the DL4J framework</description>

<properties>
    <!-- <dl4j-master.version>1.0.0-M2.1</dl4j-master.version> -->
    <dl4j-master.version>1.0.0-SNAPSHOT</dl4j-master.version>
    
    <!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
    <!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
    <nd4j.backend>nd4j-native</nd4j.backend>
    <java.version>1.8</java.version>
    <maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
    <maven.minimum.version>3.3.1</maven.minimum.version>
    <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
    <maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
    <jcommon.version>1.0.23</jcommon.version>
    <jfreechart.version>1.0.13</jfreechart.version>
    <logback.version>1.1.7</logback.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <junit.version>5.8.0-M1</junit.version>
    <javacv.version>1.5.5</javacv.version>

</properties>


<repositories>
    <repository>
        <id>sonatype-nexus-snapshots</id>
        <name>Sonatype Nexus Snapshots</name>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
        </snapshots>
    </repository>



</repositories>

<dependencies>
    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>${nd4j.backend}</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>

    <dependency>
        <groupId>org.datavec</groupId>
        <artifactId>datavec-api</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <dependency>
        <groupId>org.datavec</groupId>
        <artifactId>datavec-data-image</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <dependency>
        <groupId>org.datavec</groupId>
        <artifactId>datavec-local</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-datasets</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-core</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>resources</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-ui</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-zoo</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <!-- ParallelWrapper & ParallelInference live here -->
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-parallel-wrapper</artifactId>
        <version>${dl4j-master.version}</version>
    </dependency>
    <!-- Used in the feedforward/classification/MLP* and feedforward/regression/RegressionMathFunctions example -->
    <dependency>
        <groupId>jfree</groupId>
        <artifactId>jfreechart</artifactId>
        <version>${jfreechart.version}</version>
    </dependency>
    <dependency>
        <groupId>org.jfree</groupId>
        <artifactId>jcommon</artifactId>
        <version>${jcommon.version}</version>
    </dependency>
    <!-- Used for downloading data in some of the examples -->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.3.5</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>${logback.version}</version>
    </dependency>

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv-platform</artifactId>
        <version>${javacv.version}</version>
    </dependency>

    <!-- Test dependency. Ignore for your own application. -->
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-engine</artifactId>
        <version>${junit.version}</version>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>${junit.version}</version>
        <scope>test</scope>
    </dependency>



</dependencies>
<!-- Maven Enforcer: Ensures user has an up to date version of Maven before building -->
<build>
    <plugins>

        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>3.0.0-M5</version>
            <inherited>true</inherited>
            <dependencies>
                <dependency>
                    <groupId>org.apache.maven.surefire</groupId>
                    <artifactId>surefire-junit-platform</artifactId>
                    <version>3.0.0-M5</version>
                </dependency>
            </dependencies>
        </plugin>

        <plugin>
            <artifactId>maven-enforcer-plugin</artifactId>
            <version>1.0.1</version>
            <executions>
                <execution>
                    <id>enforce-default</id>
                    <goals>
                        <goal>enforce</goal>
                    </goals>
                    <configuration>
                        <rules>
                            <requireMavenVersion>
                                <version>[${maven.minimum.version},)</version>
                                <message>********** Minimum Maven Version is ${maven.minimum.version}. Please upgrade Maven before continuing (run "mvn --version" to check). **********</message>
                            </requireMavenVersion>
                        </rules>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>${maven-compiler-plugin.version}</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>${exec-maven-plugin.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <executable>java</executable>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>${maven-shade-plugin.version}</version>
            <configuration>
                <shadedArtifactAttached>true</shadedArtifactAttached>
                <shadedClassifierName>${shadedClassifier}</shadedClassifierName>
                <createDependencyReducedPom>true</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>org/datanucleus/**</exclude>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>reference.conf</resource>
                            </transformer>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.eclipse.m2e</groupId>
                <artifactId>lifecycle-mapping</artifactId>
                <version>1.0.0</version>
                <configuration>
                    <lifecycleMappingMetadata>
                        <pluginExecutions>
                            <pluginExecution>
                                <pluginExecutionFilter>
                                    <groupId>com.lewisd</groupId>
                                    <artifactId>lint-maven-plugin</artifactId>
                                    <versionRange>[0.0.11,)</versionRange>
                                    <goals>
                                        <goal>check</goal>
                                    </goals>
                                </pluginExecutionFilter>
                                <action>
                                    <ignore/>
                                </action>
                            </pluginExecution>
                        </pluginExecutions>
                    </lifecycleMappingMetadata>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

The error is:

agibsonccc · April 11, 2023, 7:17am

@yaoyao when did you clone this code? According to your stack trace, you have a missing file property
here:

github.com

deeplearning4j/deeplearning4j/blob/30edcc6d2a8149265e0da203380188a4c7a0bd5f/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/factory/Nd4j.java#L5257


      
          

          Class<? extends BasicMemoryManager> memoryManagerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(MEMORY_MANAGER));

          

          allowsOrder = backend.allowsOrder();

          String rand = pp.toString(RANDOM_PROVIDER, DefaultRandom.class.getName());

          Class<? extends org.nd4j.linalg.api.rng.Random> randomClazz = ND4JClassLoading.loadClassByName(rand);

          randomFactory = new RandomFactory(randomClazz);

          Class<? extends DeviceIDProvider> deviceIDProviderClass = ND4JClassLoading

                  .loadClassByName(pp.toString(DEVICE_ID_PROVDER_KEY));

          DEVICE_ID_PROVIDER = deviceIDProviderClass.newInstance();

          

          Class<? extends MemoryWorkspaceManager> workspaceManagerClazz = ND4JClassLoading

                  .loadClassByName(pp.toString(WORKSPACE_MANAGER));

          

          Class<? extends BlasWrapper> blasWrapperClazz = ND4JClassLoading.loadClassByName(pp.toString(BLAS_OPS));

          String clazzName = pp.toString(DISTRIBUTION, DefaultDistributionFactory.class.getName());

          Class<? extends DistributionFactory> distributionFactoryClazz = ND4JClassLoading.loadClassByName(clazzName);

          

          

          memoryManager = memoryManagerClazz.newInstance();

I clearly see that id in master here:

github.com

deeplearning4j/deeplearning4j/blob/41e96b765fd7ceb3cd7a80b0c42582e2440935d9/nd4j/nd4j-backends/nd4j-backend-impls/nd4j-native/src/main/resources/nd4j-native.properties#L41


      
          ndarray.order = c
          resourcemanager_state = false
          databufferfactory = org.nd4j.linalg.cpu.nativecpu.buffer.DefaultDataBufferFactory
          workspacemanager = org.nd4j.linalg.cpu.nativecpu.workspace.CpuWorkspaceManager
          alloc = javacpp
          opexec= org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner
          opexec.mode= native
          random=org.nd4j.linalg.cpu.nativecpu.rng.CpuNativeRandom
          blaslapackdelegator=org.nd4j.linalg.cpu.nativecpu.OpenblasLapackDelegator
          statsprovider=org.nd4j.linalg.cpu.nativecpu.CpuStatisticsProvider
          deviceidprovider=org.nd4j.jita.constant.DefaultDeviceIDProvider

yaoyao · April 11, 2023, 7:18am

I will clone a newest examples repo, and tri it again.

agibsonccc · April 11, 2023, 7:20am

@yaoyao no not the examples. Please quit doing anything with the examples.
If you pay attention, you’ll see I linked the source code. Not the examples.

The source code and examples are completely different repositories. If you clone the examples again you’ll just have to retrace the same steps of changing M2.1 to snapshots and you won’t accomplish anything.

yaoyao · April 11, 2023, 8:27am

I use the newest DL4J project, but the error still exists…

any other methods for fixing this ?

agibsonccc · April 11, 2023, 9:29am

@yaoyao yes by not ignoring me. I already told you what to do. I showed you literally the exact lines you should be seeing in your local version of the code.

I feel like maybe because of language barrier or something you only ready half of what I say and then just try something on your own.

It hurts that I have to guess that but you’ve been not been at all communicative in your github issues or your forum posts. I don’t even know when I solve your issue.

I’m sorry but it’s really difficult to help you if you won’t do something like ask a clarifying question.

If you can instead of ignoring me ask what I mean if that is your issue, then please do that. I would rather know what you don’t understand and then answer you then guess what your situation is.

yaoyao · April 11, 2023, 9:31am

The line number is as same as you given:

and

agibsonccc · April 11, 2023, 9:44am

@yaoyao appreciated. Please do that from now on. I’m only trying to help you. Remember I don’t have access to your machine so I don’t know what you do/don’t have.
Sorry this is frustrating for me too. Builds should not be this complicated.

We know you have the correct source code then. Could you check the .m2/repository/org/nd4j/nd4j-api/1.0.0-SNAPSHOT/ jar file to see if you find that value with the properties in there? You can also put a debugger on that line and show that to me as well.

If the correct file is there there is no reason it shouldn’t be showing up.

Could you confirm with me how you are running the program?
Edit:
@yaoyao if you are using intellij could you also run file → invalidate caches and restart?

yaoyao · April 11, 2023, 10:58am

I see the content of the generated SNAPSHOT jar file, it includes org dir and a nd4j-native.properties file. I discover that it losts the last 3 lines.

Maybe I need to build the DL4J project again?

I run the example by command mvn exec:java -Dexec.mainClass=“org.deeplearning4j.examples.quickstart.modeling.convolution.LeNetMNIST”

agibsonccc · April 11, 2023, 11:12am

@yaoyao ok so your local jar is still off somehow. Thank you for confirming. That’s why I asked you to rebuild everything as well as delete your .m2.

Rebuild the deeplearning4j project (not the examples) and make sure the modules are correct in the following places:

nd4j-api/target directory (this is where the jar file is generated and copied from to the .m2)
the .m2 folder (this is where the examples as a separate project will look for the nd4j-api module)

yaoyao · April 19, 2023, 12:21am

Hi! Followed by your instructions, I cleaned all the m2 repo and execute the mvn clean install -DskipTests in the root dir of deeplearning4j. And a 1.0.0-SNAPSHOT dir with a snapshot jar package is generated.

The null pointer error still reported.

BTW, the nd4j-native.properties file in snapshot is as same as the file in M2.1. What can I do then?

agibsonccc · April 19, 2023, 12:24am

@yaoyao I still think your problem is very localized to you somehow. I work with customers who work with snapshots all day + my own constant updating of the code and never run in to this.

Could you just run this through the debugger and see what file it tries to load? There’s zero reason this should be happening.

I’ve already showed you why it happens, where the file is. We need to break this down.

yaoyao · April 19, 2023, 12:28am

After mvn clean install -DskipTests command, the nd4j-api/target folder looks like this:

and my m2 repo is in the correct location.

Fine, I will debug it.

agibsonccc · April 19, 2023, 12:29am

@yaoyao sorry when I mean “debug” I think you misunderstood maybe? I mean run your code under the debugger where it tries to load the file and show me that. I don’t mean “just debug this yourself” I really just need more information.

yaoyao · April 21, 2023, 12:39pm

Thanks for your help, I use git to clone the source code (branch) in version 1.0.0-M2.1, and I can successfully compiled & use the corresponding snapshot it in my project.

But the source code in the master branch still occrus errors somehow. Anyway, thanks!

Regards,

DL4J lover.

agibsonccc · April 21, 2023, 10:04pm

@yaoyao that’s great! Thanks for confirming. Can you tell me a bit about your platform so I can potentially fix the issue? I still feel like I’m missing something.

yaoyao · April 22, 2023, 2:22am

Of course. This is my server config:

Linux dell-PowerEdge-R740 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

BYW, my server only have Intel CPUs, no GPU devices or CUDA environment.

cmake 3.20.2
GNU Make 4.2.1
Apache Maven 3.8.8
openjdk 11.0.18 2023-01-17

agibsonccc · April 22, 2023, 5:20am

@yaoyao thanks. I’m sorry about the weirdness there. I’ve been building on prod centos 7 for customers for a while now. That’s usually the main problem config.
I’ll keep an eye on this for the release to see if something else si going on there.

Usually you’d run in to unknown libc issues or potentially linker issues. I am in the middle of a few PRs that fix some unknown symbols: Fix undefined symbols, Migrate more ints -> sd::LongType by agibsonccc · Pull Request #9963 · deeplearning4j/deeplearning4j · GitHub
I have probably 1 more batch of changes like this as follow up but beyond that have not encountered anything specific.

Topic		Replies	Views
Dl4j-examples - clean build failure - missing org.deeplearning4j:resources:jar:1.0.0-SNAPSHOT	11	263	January 27, 2023
ND4J compile from source fails with snapshot ND4J	6	2195	May 3, 2021
Snapshots now ran every 12 hours DL4J	9	444	May 9, 2021
Failed to build dl4j-examples, compilation error DL4J	2	243	September 9, 2022
Deeplearning4j-core .jar file size	4	748	August 24, 2021

How to use the new generated DL4J artifact in DL4J-examples project?

Related Topics