-Dorg.bytedeco.javacpp.maxbytes=11024m -Dorg.bytedeco.javacpp.maxphysicalbytes=11024m
The memory of gpu not release or the setting not work?
@SidneyLann set the GC interval using: Nd4j.getMemoryManager().setOccasionalGcFrequency(50000);
or call System.gc() in places where memory runs high. One place I would recommend is right after the model is loaded.
No help, still consume all gpu RAM and shutdown the pc after 50~70 epochs.
@SidneyLann what GPU are you trying to run on? At some point your GPU just can’t handle the workload and you need to shrink your batch size or something.
Using i3080. May not related to batch size, because it can be trained 50+ epochs. After 20+ epochs, still remains 5G gpu RAM. gpu RAM may not be released or cache is used.
@SidneyLann are you running with the latest snapshots or M2.1? Recently I changed the cache up to address another issue and I wonder if that might help the situation a bit. Could you clarify a bit more about the scenario? Please don’t make me guess.
I need to know exactly when it runs out of memory, what does/doesn’t work and what you’ve tested so far. I would also like to know if you’re measuring anything at all.
Using M2.1, SNAPSHOT can’t run, because Cannot invoke “java.lang.Class.newInstance()” because “memoryManagerClazz” is null.
It can run more than 50+ epochs, so It’s hard to tell what happen.