# Dl4j cuda 11.2 running out of memory on evaluation on ubuntu 20.04

**URL:** <https://community.konduit.ai/t/dl4j-cuda-11-2-running-out-of-memory-on-evaluation-on-ubuntu-20-04/1377>\
**Category:** DL4J\
**Created:** [May 10, 2021, 6:19pm UTC](https://community.konduit.ai/t/dl4j-cuda-11-2-running-out-of-memory-on-evaluation-on-ubuntu-20-04/1377 "2021-05-10T18:19:26Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![ajmakoni](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/ajmakoni/32/497_2.png) [@ajmakoni](https://community.konduit.ai/u/ajmakoni)\
**Post date:** [May 13, 2021, 8:19am UTC](https://community.konduit.ai/t/dl4j-cuda-11-2-running-out-of-memory-on-evaluation-on-ubuntu-20-04/1377/12 "2021-05-13T08:19:59Z")

</div>

Hi guys,

To be a little more clear on the steps that we did, here are the details below:

We are using these memory settings:

> [@ajmakoni](#):
>
> -Xms2G -Xmx9G -Dorg.bytedeco.javacpp.maxbytes=11G -XX:+UseG1GC -Dorg.bytedeco.javacpp.maxphysicalbytes=12G -verbose:gc

When we enable the automatic gc, we don’t get error in the first epoch evaluation

> [@ajmakoni](#):
>
> ```auto
> Nd4j.getMemoryManager().togglePeriodicGc(true);
> Nd4j.getMemoryManager().setAutoGcWindow(5000);
> 
> ```

And if we use this custom class, the train do 8 epochs without problems

> [@ajmakoni](#):
>
> ```auto
> var xx = new IterationTerminationCondition() {
> int counter = 0;
> 			
> @Override
> public boolean terminate(double lastMiniBatchScore) {
> 				
> if(counter == 5) {
> LOGGER.info("GC start.");
> System.gc();
> LOGGER.info("GC END.");
> counter = 0;
> }
> 				
> counter++;
> return false;
> }
> 			
> @Override
> public void initialize() {				
> }
> };
> 
> ```

> [@ajmakoni](#):
>
> ```auto
> 11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Total epochs: 8
> 11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Best epoch number: 2
> 11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Score at best epoch: 0.09629346433095051
> 11:30:28.400 [main] INFO com.sample.ui.TrainProgram - Evaluate model at iteration 0 ....
> 
> ```

But it is failing here

> [@ajmakoni](#):
>
> `System.gc(); Evaluation eval = graph.evaluate(testIterator);`

> [@ajmakoni](#):
>
> ```auto
> Exception in thread "main" java.lang.RuntimeException: Error during neural network forward pass
> at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2526)
> at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4234)
> at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4189)
> at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4147)
> at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluation(ComputationGraph.java:4134)
> at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3979)
> at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3947)
> at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3926)
> at com.sample.ui.TrainProgram.evalOn(TrainProgram.java:202)
> at com.sample.ui.TrainProgram.main(TrainProgram.java:175)
> Caused by: java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (7080M + 7074M) > maxBytes (11264M)
> at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:696)
> at org.deeplearning4j.cuda.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:129)
> at org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.preOutput(CudnnConvolutionHelper.java:544)
> at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.preOutput(ConvolutionLayer.java:425)
> at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.activate(ConvolutionLayer.java:522)
> at org.deeplearning4j.nn.layers.FrozenLayer.activate(FrozenLayer.java:82)
> at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:110)
> at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2416)
> ... 9 more
> 
> ```

For now we are catching this error and save the model, but we are testing the same code on Windows with cuda 10.2 and video card GTX 1080 ti and we dont have this problem. Also you suggested:

> You can also try setting the gc period to longer.

are you able to explain more on how to achieve this?  
Thank you for your continued and timely assistance

---

_[View the full topic](https://community.konduit.ai/t/dl4j-cuda-11-2-running-out-of-memory-on-evaluation-on-ubuntu-20-04/1377)._