Arbiter: Cannot perform evaluation with NaNs present in predictions

DL4J
1

arbiter trainning throw below exception in some epochs, but arbiter trainning can be done successfully. this exception is normal?

java.lang.IllegalStateException: Cannot perform evaluation with NaNs present in predictions: 24576 NaNs present in predictions INDArray
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:286)
at org.nd4j.evaluation.classification.Evaluation.eval(Evaluation.java:403)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4192)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4131)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4089)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3938)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3900)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3878)
at org.deeplearning4j.arbiter.scoring.impl.EvaluationScoreFunction.score(EvaluationScoreFunction.java:78)
at org.deeplearning4j.arbiter.scoring.impl.BaseNetScoreFunction.score(BaseNetScoreFunction.java:79)
at org.deeplearning4j.arbiter.scoring.impl.BaseNetScoreFunction.score(BaseNetScoreFunction.java:59)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.callHelper(ComputationGraphTaskCreator.java:239)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.call(ComputationGraphTaskCreator.java:133)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.call(ComputationGraphTaskCreator.java:88)
at org.nd4j.shade.guava.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.nd4j.shade.guava.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.nd4j.shade.guava.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

3

Only tanh can be used for LSTM, Other activation function should cause above exception.

4

@SidneyLann I’m not quite convinced it’s a simple “yes/no this is an absolute rule” you can use a a few different activation functions with LSTMs.

If you see NANs that’s generally just from an unstable network though. This has the same trade offs as training any neural network.

5

It has no error in trainning stage, just in validating stage has error.