Nd4j memory leaks

Hello again, I have tried to use the workspace feature in order to solve my memory leak problem.
I’m running my whole application inside a loop, each iteration creates a workspace (with the same id). After each iteration I check the Pointer.availablePhysicalBytes() and I see that it always decreases.
I’m running inside a Linux container using the CPU backend.
I have a total of 3GB for the container, 250Mb for the heap, the rest goes to the off heap.
I gave the ws min, initial and max size of 2GB, without spill. I can also see that the workspace offset resets correctly and that it doesn’t go above ~300Mb at most.
Any ideas where is the memory leak coming from?

@Tal can we look at code? In know in M2.1 lots of references build up in the deallocator service which affects memory usage. I’d need more details about the error, how you’re measuring it and what you’re seeing. Like is it a javacpp off heap? Manually measured memory usage?

Hi :slight_smile:
I don’t have the code available right now (I’ll add it later if it will be needed), but basically I used arrange twice and then created a meshgrid from that. Did that in a loop and I saw the memory usage go up, even when GC is called or workspace is used. I measured it by using javacpp physical bytes, and also by checking the docker container’s memory usage.

@Tal I still need the code to help you here. “Leaks” need to be measured in order to be fixed. This could be JVM memory, off heap memory. WIthout profiling and just a vague notion of “memory going up” I can’t really help you here.