Memory leak in ND4J if using the ZGC garbage collector

ND4J
1

I know that ZGC is still in an experimental phase, but recently I tried it out and it caused a memory leak.
here are the logs:

java.lang.OutOfMemoryError: Cannot allocate new LongPointer(1): totalBytes = 512, physicalBytes = 4146M
        at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.<init>(Lwjgl3Application.java:122)
        at com.flow.app.lwjgl3.Lwjgl3Launcher.createApplication(Lwjgl3Launcher.java:33)
        at com.flow.app.lwjgl3.Lwjgl3Launcher.main(Lwjgl3Launcher.java:19)
Caused by: java.lang.OutOfMemoryError: Cannot allocate new LongPointer(1): totalBytes = 512, physicalBytes = 4146M
        at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:88)
        at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:53)
        at org.nd4j.linalg.cpu.nativecpu.ops.CpuOpContext.setIArguments(CpuOpContext.java:60)
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1558)
        at org.nd4j.linalg.factory.Nd4j.exec(Nd4j.java:6566)
        at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.concat(CpuNDArrayFactory.java:579)
        at org.nd4j.linalg.factory.Nd4j.concat(Nd4j.java:4857)
        at org.nd4j.linalg.factory.BaseNDArrayFactory.vstack(BaseNDArrayFactory.java:983)
        at org.nd4j.linalg.factory.Nd4j.vstack(Nd4j.java:4641)
        at com.flow.vision.Preprocess.eulerAnglesToRotationMatrix(Preprocess.java:161)
        at com.flow.app.StreamScreen.updateLanes(StreamScreen.java:129)
        at com.flow.app.StreamScreen.render(StreamScreen.java:183)
        at com.badlogic.gdx.Game.render(Game.java:46)
        at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Window.update(Lwjgl3Window.java:403)
        at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.loop(Lwjgl3Application.java:143)
        at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.<init>(Lwjgl3Application.java:116)
        ... 2 more
Caused by: java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (4146M) > maxPhysicalBytes (4040M)
        at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:700)
        at org.bytedeco.javacpp.Pointer.init(Pointer.java:126)
        at org.bytedeco.javacpp.LongPointer.allocateArray(Native Method)
        at org.bytedeco.javacpp.LongPointer.<init>(LongPointer.java:80)
        ... 17 more

specs: windows 10, nd4j 1.0.0-M1

thanks!

2

It’s not a memory leak. Using JavaCPP 1.5.6 should help with this: OutOfMemoryError: Physical memory usage is too high: physicalBytes (5300M) > maxPhysicalBytes (5300M) · Issue #468 · bytedeco/javacpp · GitHub

1 Like
3

Thanks for the reply ! This error showed up after running the app for 3+ mins with ZGC. It works fine if not using ZGC with almost constant memory usage. The other thing I noticed was extremely high RAM usage in the profiler when ZGC was used. I will post screenshots soon.

btw how do I update javaCPP ? I only have nd4j as a dependency. Do I need to edit the nd4j pom files for this ?

4

No ZGC:

no_zgc
no_zgc1338Ă—852 113 KB

5

With ZGC:

zgc2
zgc21306Ă—725 86.3 KB

In this case the heap shoots to 2GB. and the app finally crrashes at last.

Let me know if you need any heap dumps or other things!
thx

6

Yes, in the pom.xml file like this: https://github.com/bytedeco/javacpp#downloads

1 Like
7

It’s actually the nature of ZGC. It provides low-latency garbage collection, but at a price - by “eating” more RAM. It depends on many factors of course, but I’ve noticed in practice, that when the heap is quite volatile and a lot of stuff needs to be collected by GC very often, ZGC starts taking more memory for own purposes in order to guarantee the low-latency (it needs memory space in order keep all the “household” info). There’s one thing though - the actual array memory allocations for DL4J occur in the native memory, not heap. So what you see on your heap usage graphs is the part which is being used basically by your app directly (like by generated/loaded test/train data, service data etc.). Those graphs won’t show you how much exactly the whole application takes - this thing can be done by running org.bytedeco.javacpp.Pointer#physicalBytes()

1 Like