# Releasing memory from Datavec Local?

**URL:** <https://community.konduit.ai/t/releasing-memory-from-datavec-local/246>\
**Category:** DataVec\
**Created:** [March 9, 2020, 11:42pm UTC](https://community.konduit.ai/t/releasing-memory-from-datavec-local/246 "2020-03-09T23:42:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![RobertJSharp](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/robertjsharp/32/82_2.png) [@RobertJSharp](https://community.konduit.ai/u/RobertJSharp)\
**Post date:** [March 9, 2020, 11:42pm UTC](https://community.konduit.ai/t/releasing-memory-from-datavec-local/246/1 "2020-03-09T23:42:54Z")

</div>

Hello All,

I’ve set up a mini-server that is constantly performing inferences very quickly in real time, however I also need it to perform a transform on a small amount of data. The LocalTransformExecutor is quite a bit faster than Spark (Local), however I’m finding that it doesn’t release it’s memory. Running LocalTransformExecutor.execute() over and over I eventually get:  
org.apache.arrow.memory.OutOfMemoryException: Failure allocating buffer.  
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:56)  
at org.apache.arrow.memory.AllocationManager.(AllocationManager.java:90)

This is easily tested simply putting an execute in a loop. It won’t release until you exit the jvm. If I pause the data collection it won’t gc, and if put in a System.gc() it does nothing. I’ve tried many of the Nd4j static memory management commands to try to release the allocations, purgeCache etc, even tried to set up workspaces. I know that workspaces has been implemented and fixed similar problems with train/inferencing, but I don’t think it’s set up on Local Datavec. From what I can tell, it seems to be arrow holding onto references, but I’m not sure.

Does any know any simple way to get it to release it’s memory, or am I waiting for fix here (or trying to fix it myself)?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![saudet](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/saudet/32/69_2.png) [@saudet](https://community.konduit.ai/u/saudet)\
**Post date:** [March 10, 2020, 7:38am UTC](https://community.konduit.ai/t/releasing-memory-from-datavec-local/246/2 "2020-03-10T07:38:39Z")

</div>

Could you provide a small code snippet that exhibits this behavior? This sounds like a bug though, in which case we may want to track this as a GitHub issue.

---

<div class="post-metadata">

**Author:** ![RobertJSharp](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/robertjsharp/32/82_2.png) [@RobertJSharp](https://community.konduit.ai/u/RobertJSharp)\
**Post date:** [March 10, 2020, 7:52am UTC](https://community.konduit.ai/t/releasing-memory-from-datavec-local/246/3 "2020-03-10T07:52:57Z")

</div>

I tested with a very cut down version of org.datavec.transform.basic.BasicDataVecExampleLocal.

Modified to a simple transform:

```
    TransformProcess tp = new TransformProcess.Builder(inputDataSchema)
    		.addConstantDoubleColumn("Test-Column", 20)
		.build();

```

and then loop the .execute:

```
    List<List<Writable>> processedData = null;
    for (int i=0;i<100000;i++) {
    	processedData = LocalTransformExecutor.execute(originalData, tp);
    	/* Any memory technique I could think of here */
    	System.gc();
    }
  	System.gc();

```

still doesn’t let go of the memory until it exits.

Thanks for the reply.

---

<div class="post-metadata">

**Author:** ![saudet](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/saudet/32/69_2.png) [@saudet](https://community.konduit.ai/u/saudet)\
**Post date:** [March 13, 2020, 1:22am UTC](https://community.konduit.ai/t/releasing-memory-from-datavec-local/246/4 "2020-03-13T01:22:22Z")

</div>

That does indeed look like a bug in DataVec. @AlexBlack is probably the one that knows that code best, but in any case, please open an issue on GitHub for this. Thanks!