Dl4j cuda 11.2 running out of memory on evaluation on ubuntu 20.04

[ajmakoni]

Hi guys,

To be a little more clear on the steps that we did, here are the details below:

We are using these memory settings:

@ajmakoni

-Xms2G -Xmx9G -Dorg.bytedeco.javacpp.maxbytes=11G -XX:+UseG1GC -Dorg.bytedeco.javacpp.maxphysicalbytes=12G -verbose:gc

When we enable the automatic gc, we don’t get error in the first epoch evaluation

@ajmakoni

Nd4j.getMemoryManager().togglePeriodicGc(true);
Nd4j.getMemoryManager().setAutoGcWindow(5000);

And if we use this custom class, the train do 8 epochs without problems

@ajmakoni

var xx = new IterationTerminationCondition() {
			int counter = 0;
			
			@Override
			public boolean terminate(double lastMiniBatchScore) {
				
				if(counter == 5) {
					LOGGER.info("GC start.");
					System.gc();
					LOGGER.info("GC END.");
					counter = 0;
				}
				
				counter++;
				return false;
			}
			
			@Override
			public void initialize() {				
			}
		};

@ajmakoni

11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Total epochs: 8
11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Best epoch number: 2
11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Score at best epoch: 0.09629346433095051
11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Evaluate model at iteration 0 ....

But it is failing here

@ajmakoni

System.gc(); Evaluation eval = graph.evaluate(testIterator);

@ajmakoni

Exception in thread "main" java.lang.RuntimeException: Error during neural network forward pass
	at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2526)
	at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4234)
	at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4189)
	at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4147)
	at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4134)
	at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3979)
	at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3947)
	at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3926)
	at com.sample.ui.TrainProgram.evalOn(TrainProgram.java:202)
	at com.sample.ui.TrainProgram.main(TrainProgram.java:175)
Caused by: java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (7080M + 7074M) > maxBytes (11264M)
	at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:696)
	at org.deeplearning4j.cuda.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:129)
	at org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.preOutput(CudnnConvolutionHelper.java:544)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.preOutput(ConvolutionLayer.java:425)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.activate(ConvolutionLayer.java:522)
	at org.deeplearning4j.nn.layers.FrozenLayer.activate(FrozenLayer.java:82)
	at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:110)
	at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2416)
	... 9 more

For now we are catching this error and save the model, but we are testing the same code on Windows with cuda 10.2 and video card GTX 1080 ti and we dont have this problem. Also you suggested:

You can also try setting the gc period to longer.

are you able to explain more on how to achieve this?
Thank you for your continued and timely assistance