So we have a simple toy application (loads some word vectors, does some inference), and we’ve noticed that physical memory usage is increasing. We believe that there is some memory leak occuring. We’re wondering how we would go about debugging it. We want to use tools like valgrind, but there really isn’t much documentation on what the best practices are. We think the error is possibly coming from JavaCPP but we’d like your input on how to diagnose the problem.
At a high level, we notice that through JMX, we see the heap and off heap memory stay stable. But the system memory for the Java process appears to increase in output of the
top program. We have identified that it does occur when we start inference. The other parts of the application don’t seem to cause it. As we continue to do inference, it will eventually throw an OutOfMemoryError. Its clear that this is some native memory issue.
To be specific, the javaCPP physicalBytes appears to be increasing and eventually exceeds the javaCPP maxPhysicalBytes parameter. Essentially we want to identify the source of this issue and fix it. Could it be that the deallocator is not running as often as it should? Is it not fast enough at deallocating memory perhaps? We’re not really sure how we can answer these questions. I’m hoping you guys have any insights on how we can debug this issue.