I get NaN on arm server when training / predicting using dl4j 1.0.0.-SNAPSHOT compiled from GitHub - KonduitAI/deeplearning4j: Eclipse Deeplearning4j, ND4J, DataVec and more - deep learning & linear algebra for Java/Scala with GPUs + Spark . The enviroment is same with Run on ARM cpu server and Comp…

OK. NaN on arm server opened 07:52AM - 30 Mar 20 UTC (UTC) closed 09:02AM - 28 Dec 20 UTC (UTC) [liweigu] liweigu Bug …

Add this code Nd4j.getExecutioner().setProfilingMode(OpExecutioner.ProfilingMode.NAN_PANIC); and get the error infomation: Warning: Versions of org.bytedeco:javacpp:1.5.2 and org.bytedeco:openblas:0.3.9-1.5.3-SNAPSHOT do not match. [WARNING] org.nd4j.linalg.exception.ND4JOpProfilerException: …

After changing version 1.5.2 to 1.5.3, i get new error: [WARNING] org.nd4j.linalg.exception.ND4JOpProfilerException: P.A.N.I.C.! Op.Z() contains 30 NaN value(s): at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForNaN (OpExecutionerUtil.java:61) at org.nd4j.linalg.api.ops.executione…

@liweigu does this happen with the same model on an x86 box?

[image] liweigu: Here’s a simple program that caculate the sum of x and y. It can predict the sum result (x + y = [[0.5550]]) on windows using dl4j beta6, @agibsonccc I guess the answer to your question is yes.

It takes time to verify that, i’ll reply after the test. But it’s ok on windows x86 for beta6.

When building nd4j (in GitHub - KonduitAI/deeplearning4j: Eclipse Deeplearning4j, ND4J, DataVec and more - deep learning & linear algebra for Java/Scala with GPUs + Spark ) with -Djavacpp.platform=linux-x86_64 on cpu server, i get error [ERROR] Failed to execute goal on project nd4j-tensorflow: Coul…

We need to build with the --update-snapshots option as well, like mvn --update-snapshots -Djavacpp.platform=linux-x86_64 clean ...

It can compile with --update-snapshots -Djavacpp.platform=linux-x86_64 . And the testing program can train and predict with no NaN, it can print the result: x + y = [[0.5548]] It seems the NaN only occurs on arm server. @treo @agibsonccc

Very interesting. Can you write simple test for matrix multiplication, or use one of our tests in order to check what’s going on there? I.e. this test: public void testMMul() { INDArray arr = Nd4j.create(new double[][] {{1, 2, 3}, {4, 5, 6}}); INDArray assertion = Nd4j.create…

@raver119 The nd4j test is ok, it can print test’s value right. assertion = [[ 14.0000, 32.0000], [ 32.0000, 77.0000]] test = [[ 14.0000, 32.0000], [ 32.0000, 77.0000]] ( I changed the code to System.out.println("test = " + test); ) But my program still gets ND4JOpProfilerExce…

NaN on arm server

DL4J

treo March 30, 2020, 7:02am 13

From Run on ARM cpu server - #6 by liweigu

Topic		Replies	Views
Run on ARM cpu server DL4J	6	1484	February 26, 2020
NaNs present in prediction DL4J	3	1432	March 23, 2021
Simple CNN predicts NaNs DL4J	3	542	November 2, 2020
JsonMappingException ND4J	35	1760	December 4, 2020
Compiling on ARM ND4J	29	4001	April 7, 2021

NaN on arm server

Related topics