Hi. While training a SameDiff model I get an exception if I try to increase my batch size:
Exception in thread "main" java.lang.IllegalStateException: Cannot remove array from ArrayStore: no array with this length exists in the cache
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:268)
at org.nd4j.autodiff.samediff.internal.memory.ArrayCacheMemoryMgr$ArrayStore.removeObject(ArrayCacheMemoryMgr.java:255)
at org.nd4j.autodiff.samediff.internal.memory.ArrayCacheMemoryMgr$ArrayStore.access$100(ArrayCacheMemoryMgr.java:185)
at org.nd4j.autodiff.samediff.internal.memory.ArrayCacheMemoryMgr.release(ArrayCacheMemoryMgr.java:146)
at org.nd4j.autodiff.samediff.internal.InferenceSession.getOutputs(InferenceSession.java:345)
at org.nd4j.autodiff.samediff.internal.TrainingSession.getOutputs(TrainingSession.java:149)
at org.nd4j.autodiff.samediff.internal.TrainingSession.getOutputs(TrainingSession.java:31)
at org.nd4j.autodiff.samediff.internal.AbstractSession.output(AbstractSession.java:391)
at org.nd4j.autodiff.samediff.internal.TrainingSession.trainingIteration(TrainingSession.java:115)
at org.nd4j.autodiff.samediff.SameDiff.fitHelper(SameDiff.java:1713)
at org.nd4j.autodiff.samediff.SameDiff.fit(SameDiff.java:1569)
at org.nd4j.autodiff.samediff.SameDiff.fit(SameDiff.java:1509)
at org.nd4j.autodiff.samediff.config.FitConfig.exec(FitConfig.java:173)
My model is a sort of Transformer’s encoder and the main part of it is self-attention ( MultiHeadDotProductAttention
). Having the batch size of 120 and sequence length of 64 with embedding size (hidden size) of 768 was working fine. But after increasing the batch size to 140 (or more) or the sequence length to 128 (or more) I got the failure. My aim is to have at least 256 sequences in a batch and a sequence length of 128. I also have the following settings for a frequent GC cleanup, but the error is triggered even without them :
getMemoryManager().setAutoGcWindow(10000);
getMemoryManager().togglePeriodicGc(true);