I’m running into an issue where Nd4j is generating errors during maven unit test execution. I can’t trace the error as it only happens when during mvn install, not when running the unit tests from the IDE.
NullPointer d1 is marked non-null but is null
I can’t see the full error as it only pops during mvn install, but I googled “Nd4j d1” and was taken straight to the distance transforms docs page. That makes sense because I am using the allEuclideanDistances transform.
@wcneill There are NonNull annotation processors that will throw a null pointer exception during argument validation. Could you elaborate more on what you’re trying to do?
Well, I’m just trying to build a branch of a project I’m responsible for at work.
I’ve written comprehensive unit tests that all pass in IDE, but when I check the output of mvn clean install I see that there are two test failures. Unfortunately, maven does not include a full stack trace in its output. Even using verbose mode, all I get is this:
java.lang.NullPointerException: d1 is marked non-null but is null
It won’t even tell me where the error originates from except that it happens in a specific unit test.
The only reason I suspect it’s Nd4j is that it’s the only library we use that we didn’t roll ourselves, and googling that error shows it comes from some Lombok annotations, which we don’t use.
Unfortunately, I cannot supply the code but I used Nd4j to implement a few clustering algorithms, and specifically make use of Transforms.allEuclideanDistances which has arguments d1 and d2 in the docs.
I’m sorry to be so hand-wavy… Tell me what information might be useful to you besides supplying actual code and I’ll see if I can get it to you.
@wcneill thanks a lot. That’s odd. Did you verify that the arrays being passed in to the nd4j function calls aren’t null? Like I said those are just automatic null pointers that get thrown when arguments are null. I can only guess that those one of the input arrays there is null.
In general we would need to try to isolate this to figure out if there is a problem. I would recommend putting a debugger on the function or dissecting your code base using Find Usages… to see what nd4j calls are there.
There is definitely a null input for some reason and that should be fixed.
@wcneill eclipse and intellij both use the same protocol for setting up a debugger. Generally you use a “connect remotely” setup in intelilj. Setting breakpoints should be able to happen anywhere. See more here: Tutorial: Remote debug | IntelliJ IDEA
When you get your workflow setup I’m happy to help if there is a bug or something.
I got the remote maven debugger up and running. I never did find any null values and it is still unclear as to why the error isn’t thrown during in IDE unit testing, but I think I’ve cleared up the issue.
When performing some operations, I needed to convert a similarity matrix to a distance matrix, so I used similarity.rdiv(1). As my similarity matrix has a 0 valued diagonal, the distance matrix ended up with infinity values on the diagonal. This was causing some arrays full of NaNs to be passed around while using Transformers.allEuclideanDistances.
I implemented a spin off of rdiv resulting in something similar to calling np.divide(num, denom, out, where=(denom != 0), which seems to have cleared up the error I was seeing during maven tests.
@wcneill if you could ever reproduce this I would still love to solve this. I hate mysteries like that. It could have been a secondary exception due to the Infs there. What java version was this? Maybe we could reproduce this testing on newer java verrsions.