# NaN on arm server **URL:** \ **Category:** DL4J\ **Created:** [March 27, 2020, 2:08am UTC](https://community.konduit.ai/t/nan-on-arm-server/320 "2020-03-27T02:08:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 27, 2020, 2:08am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/1 "2020-03-27T02:08:30Z")
I get NaN on arm server when training / predicting using dl4j 1.0.0.-SNAPSHOT compiled from [GitHub - KonduitAI/deeplearning4j: Eclipse Deeplearning4j, ND4J, DataVec and more - deep learning & linear algebra for Java/Scala with GPUs + Spark](https://github.com/KonduitAI/deeplearning4j). The enviroment is same with [Run on ARM cpu server](https://community.konduit.ai/t/run-on-arm-cpu-server/175) and [Compiling on ARM](https://community.konduit.ai/t/compiling-on-arm/283) . Here’s a simple program that caculate the sum of x and y. It can predict the sum result (x + y = [[0.5550]]) on windows using dl4j beta6, and predict NaN on arm server using dl4j 1.0.0.-SNAPSHOT. > ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 27, 2020, 3:00am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/2 "2020-03-27T03:00:25Z")
Add this code > Nd4j.getExecutioner().setProfilingMode(OpExecutioner.ProfilingMode.NAN\_PANIC); and get the error infomation: Warning: Versions of org.bytedeco:javacpp:1.5.2 and org.bytedeco:openblas:0.3.9-1.5.3-SNAPSHOT do not match. [WARNING] org.nd4j.linalg.exception.ND4JOpProfilerException: P.A.N.I.C.! Op.Z() contains 30 NaN value(s): at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForNaN (OpExecutionerUtil.java:61) at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForAny (OpExecutionerUtil.java:65) at org.nd4j.linalg.api.blas.impl.BaseLevel3.gemm (BaseLevel3.java:77) at org.nd4j.linalg.api.ndarray.BaseNDArray.mmuli (BaseNDArray.java:3156) at org.deeplearning4j.nn.layers.BaseLayer.preOutputWithPreNorm (BaseLayer.java:316) ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 27, 2020, 3:05am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/3 "2020-03-27T03:05:42Z")
After changing version 1.5.2 to 1.5.3, i get new error: [WARNING] org.nd4j.linalg.exception.ND4JOpProfilerException: P.A.N.I.C.! Op.Z() contains 30 NaN value(s): at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForNaN (OpExecutionerUtil.java:61) at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForAny (OpExecutionerUtil.java:65) at org.nd4j.linalg.api.blas.impl.BaseLevel3.gemm (BaseLevel3.java:77) at org.nd4j.linalg.api.ndarray.BaseNDArray.mmuli (BaseNDArray.java:3156) at org.deeplearning4j.nn.layers.BaseLayer.preOutputWithPreNorm (BaseLayer.java:316) at org.deeplearning4j.nn.layers.BaseLayer.preOutput (BaseLayer.java:289) at org.deeplearning4j.nn.layers.BaseLayer.activate (BaseLayer.java:337) at org.deeplearning4j.nn.layers.AbstractLayer.activate (AbstractLayer.java:257) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.ffToLayerActivationsInWs (MultiLayerNetwork.java:1132) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore (MultiLayerNetwork.java:2750) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore (MultiLayerNetwork.java:2708) at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore (BaseOptimizer.java:170) at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize (StochasticGradientDescent.java:63) at org.deeplearning4j.optimize.Solver.optimize (Solver.java:52) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper (MultiLayerNetwork.java:2309) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit (MultiLayerNetwork.java:2267) at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit (MultiLayerNetwork.java:2330) ---
**Author:** ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)\ **Post date:** [March 27, 2020, 3:29am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/4 "2020-03-27T03:29:34Z")
@liweigu does this happen with the same model on an x86 box? ---
**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\ **Post date:** [March 27, 2020, 7:47am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/5 "2020-03-27T07:47:31Z")
> [@liweigu](#): > > Here’s a simple program that caculate the sum of x and y. It can predict the sum result (x + y = [[0.5550]]) on windows using dl4j beta6, @agibsonccc I guess the answer to your question is yes. ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 27, 2020, 9:07am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/6 "2020-03-27T09:07:35Z")
It takes time to verify that, i’ll reply after the test. But it’s ok on windows x86 for beta6. ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 28, 2020, 10:07am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/7 "2020-03-28T10:07:23Z")
When building nd4j (in [GitHub - KonduitAI/deeplearning4j: Eclipse Deeplearning4j, ND4J, DataVec and more - deep learning & linear algebra for Java/Scala with GPUs + Spark](https://github.com/KonduitAI/deeplearning4j.git)) with -Djavacpp.platform=linux-x86\_64 on cpu server, i get error [ERROR] Failed to execute goal on project nd4j-tensorflow: Could not resolve dependencies for project org.nd4j:nd4j-tensorflow:jar:1.0.0-SNAPSHOT: Failure to find org.bytedeco:tensorflow:jar:linux-x86\_64:1.15.2-1.5.3-20200324.074704-327 in [https://oss.sonatype.org/content/repositories/snapshots](https://oss.sonatype.org/content/repositories/snapshots) was cached in the local repository, resolution will not be reattempted until the update interval of sonatype-nexus-snapshots has elapsed or updates are forced → [Help 1] It blocks me. ---
**Author:** ![saudet](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/saudet/32/69_2.png) [@saudet](https://community.konduit.ai/u/saudet)\ **Post date:** [March 28, 2020, 2:46pm UTC](https://community.konduit.ai/t/nan-on-arm-server/320/8 "2020-03-28T14:46:33Z")
We need to build with the --update-snapshots option as well, like `mvn --update-snapshots -Djavacpp.platform=linux-x86_64 clean ...` ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 30, 2020, 3:07am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/9 "2020-03-30T03:07:23Z")
It can compile with --update-snapshots -Djavacpp.platform=linux-x86\_64 . And the testing program can train and predict with no NaN, it can print the result: x + y = [[0.5548]] It seems the NaN only occurs on arm server. @treo @agibsonccc ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 4:28am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/10 "2020-03-30T04:28:44Z")
Very interesting. Can you write simple test for matrix multiplication, or use one of our tests in order to check what’s going on there? I.e. this test: ```auto public void testMMul() { INDArray arr = Nd4j.create(new double[][] {{1, 2, 3}, {4, 5, 6}}); INDArray assertion = Nd4j.create(new double[][] {{14, 32}, {32, 77}}); INDArray test = arr.mmul(arr.transpose()); assertEquals(getFailureMessage(), assertion, test); } ``` ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 30, 2020, 6:56am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/11 "2020-03-30T06:56:59Z")
@raver119 The nd4j test is ok, it can print test’s value right. assertion = [[14.0000, 32.0000], [32.0000, 77.0000]] test = [[14.0000, 32.0000], [32.0000, 77.0000]] ( I changed the code to System.out.println("test = " + test); ) But my program still gets ND4JOpProfilerException as above in the same machine and in the same java project. ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 6:59am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/12 "2020-03-30T06:59:39Z")
Hmmm… Are you using some cloud provider? Is there any chance we could get access to similar ARM machine? ---
**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\ **Post date:** [March 30, 2020, 7:02am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/13 "2020-03-30T07:02:59Z")
From [Run on ARM cpu server - #6 by liweigu](https://community.konduit.ai/t/run-on-arm-cpu-server/175/6) > [@Run on ARM cpu server](https://community.konduit.ai/t/run-on-arm-cpu-server/175/6): > > It’s Huawei’s server, but not from Huawei Cloud. > Handle 0x0001, DMI type 1, 27 bytes > System Information > Manufacturer: Huawei > Product Name: TaiShan 2280 V2 ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 7:03am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/14 "2020-03-30T07:03:01Z")
Also: can you please print out your input data and labels before feeding them into neural network? ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 30, 2020, 7:28am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/15 "2020-03-30T07:28:40Z")
For this line `int nEpochs = 500;` If set it to 50, there is no NaN, and it can print the result `x + y = [[0.4639]]` So the NaN doesn’t come out in the 1st epoch on training. ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 30, 2020, 7:34am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/16 "2020-03-30T07:34:21Z")
I put the data at > ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 7:39am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/17 "2020-03-30T07:39:52Z")
Hmmm… so it sounds like hardware-specific overflow then… We’ll definitely need to set up ARM machine and see what’s up there. ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 7:45am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/18 "2020-03-30T07:45:15Z")
@liweigu can you file a github issue please? We need to get to the bottom here ---
**Author:** ![liweigu](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/liweigu/32/68_2.png) [@liweigu](https://community.konduit.ai/u/liweigu)\ **Post date:** [March 30, 2020, 7:52am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/19 "2020-03-30T07:52:49Z")
OK. > > > \#### Issue Description > > Please describe our issue, along with: > \- expected beh…avior > \- encountered behavior > NaN occors in https://gist.github.com/liweigu/bfcbd2a6e612e02ad7b24fee3cfac235 for epoch=500. > It's ok for epoch=50. > The detail is at https://community.konduit.ai/t/nan-on-arm-server/320 . > > \#### Version Information > > Please indicate relevant versions, including, if relevant: > > \* Deeplearning4j version > 1.0.0-SNAPSHOT > \* Platform information (OS, etc) > arm on Huawei > \* CUDA version, if used > \* NVIDIA driver version, if in use > > \#### Additional Information > > Where applicable, please also provide: > > \* Full log or exception stack trace (ideally in a Gist: gist.github.com) > \* pom.xml file or similar (also in a Gist) > > \#### Contributing > > If you'd like to help us fix the issue by contributing some code, but would > like guidance or help in doing so, please mention it! ---
**Author:** ![raver119](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@raver119](https://community.konduit.ai/u/raver119)\ **Post date:** [March 30, 2020, 9:25am UTC](https://community.konduit.ai/t/nan-on-arm-server/320/20 "2020-03-30T09:25:56Z")
Thank you very much. We’ll check it out. [Next page](https://community.konduit.ai/t/nan-on-arm-server/320.md?page=2)