Arbiter: Cannot perform evaluation with NaNs present in predictions

arbiter trainning throw below exception in some epochs, but arbiter trainning can be done successfully. this exception is normal?

java.lang.IllegalStateException: Cannot perform evaluation with NaNs present in predictions: 24576 NaNs present in predictions INDArray
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:286)
at org.nd4j.evaluation.classification.Evaluation.eval(Evaluation.java:403)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4192)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4131)
at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4089)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3938)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3900)
at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3878)
at org.deeplearning4j.arbiter.scoring.impl.EvaluationScoreFunction.score(EvaluationScoreFunction.java:78)
at org.deeplearning4j.arbiter.scoring.impl.BaseNetScoreFunction.score(BaseNetScoreFunction.java:79)
at org.deeplearning4j.arbiter.scoring.impl.BaseNetScoreFunction.score(BaseNetScoreFunction.java:59)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.callHelper(ComputationGraphTaskCreator.java:239)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.call(ComputationGraphTaskCreator.java:133)
at org.deeplearning4j.arbiter.task.ComputationGraphTaskCreator$GraphLearningTask.call(ComputationGraphTaskCreator.java:88)
at org.nd4j.shade.guava.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.nd4j.shade.guava.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.nd4j.shade.guava.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Only tanh can be used for LSTM, Other activation function should cause above exception.

@SidneyLann I’m not quite convinced it’s a simple “yes/no this is an absolute rule” you can use a a few different activation functions with LSTMs.

If you see NANs that’s generally just from an unstable network though. This has the same trade offs as training any neural network.

It has no error in trainning stage, just in validating stage has error.