Hello again, I found out strange thing. When training TinyYOLO with pretrained weights and if all data (all batches) has same dimensions, everything is fine. But if every batch has different data dimensions (same dimensions in actual batch, but different between other batches) training score goes i…

Are you running beta6 or the latest snapshots?

I’am running beta6, cuda 10.2

I see. Can you please try the latest snapshots, so if there’s something off we could debug it based on current master rather than beta6 release?

Instructions for using snapshots can be found here: https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

Thank you guys. Tried with latest snapshots and it is working in same way. NaN at 5th iteration.

Before you start the 5th iteration, can you set Nd4j.getExecutioner().setProfilingMode(OpExecutioner.ProfilingMode.NAN_PANIC)? It will likely be very slow afterwards, because it checks if there is a NAN after each operation, but that way we can see where they appear.

I got this exception even in 0 iteration: Exception in thread “main” org.nd4j.linalg.exception.ND4JOpProfilerException: P.A.N.I.C.! Op.Z() contains 24502 NaN value(s): at org.nd4j.linalg.api.ops.executioner.OpExecutionerUtil.checkForNaN(OpExecutionerUtil.java:61) at org.nd4j.linalg.api.ops.execut…

That looks suspicious. Can you provide us with a minimal demo project that produces the NaNs? The inputs should be as close to the original as possible in shape, but it would be fine for them to be random otherwise.

There it is: TinyYOLO_cuDNN.zip Google Drive file. Of course this is not, what I am using, but it is acting similar way. I hope code will be easier to run then last time :slight_smile:. Thank you for your time.

Sorry for letting you wait so long. [image] Rolnire: I got this exception even in 0 iteration: I’ve finally gotten around to check this, and it is because the Yolo output layer actually uses NaN’s as part of the computation. This makes the search for where it goes to NaN later on quite a bit…

TinyYOLO training goes into NaN with cuDNN

DL4J

Topic		Replies	Views
TinyYolo training goes into NaN with CPU Backend DL4J	6	1020	April 16, 2020
Tiny YOLO PredictedObjects NaN DL4J	14	2191	June 18, 2024
Simple CNN predicts NaNs DL4J	3	544	November 2, 2020
NaNs present in prediction DL4J	3	1443	March 23, 2021
NaN on arm server DL4J	20	1569	December 28, 2020

TinyYOLO training goes into NaN with cuDNN

Related topics