Confusion matrix for non-binary data?

DL4J
1

I have code that creates an INDArray with 1000 predictions and I want to compare them with 1000 solutions to make a confusion matrix. Now, I already found out roughly how to do that:

Evaluation eval = new Evaluation();

Yhat.setShapeAndStride(new int[]{1,1000}, new int[]{1000,1});
Ytest.setShapeAndStride(new int[]{1,1000}, new int[]{1000,1});

eval.eval(Ytest, Yhat);
System.out.println(eval.stats());

Problem 1: you already see I had to reshape the Arrays. Originally they had shape:
Rank: 1, DataType: LONG, Offset: 0, Order: c, Shape: [1000], Stride: [1]
and the code would throw a runtime error complaining “Labels and predictions must always be rank 2 or higher”. This version doesn’t throw an error, but it’s clearly still the wrong shape: it results in a confusion matrix that thinks there are 1000 classes. What shape and “stride” should I use for the matrix, and what’s the stride for anyway?

Problem 2: I figured out that what the Evaluation wants is a matrix with 0s and 1s, where each row is a data point, the columns are potential classes, and there’s a 1 in the column that marks the correct class. However, my data is in the shape [ 5, 2, 6…] meaning the first data point is predicted to be class 5, and so on.
So, either:
a) How do I transform the matrix into a 0s and 1s matrix?
b) How do I make a confusion matrix out of a “non-binary” matrix?

2

For your case you can simply ignore the stride. I’m not even sure how you decided that you need to call .setShapeAndStride, instead of reshape.

What you’ve got here is a rank 1 matrix, also better known as a vector.

The operations you are working with expect to be working with mini-batches, where the first dimension is for the minibatch size and the second is the feature size.

Given that you’ve got 1000 predictions, you want it to have the shape [1000, 1].

You need to one-hot encode it. That means your data should have the shape [1000, numOfClasses] and then you set a 1 where ever your predicted class came from.

Given that you have data that appears to be coming from something other than DL4J, it may be easier to just implement the confusion matrix calculation yourself, instead of massaging your data to fit the format DL4J expects.

3

Well first I tried
Yhat.reshape(1,1000)
Ytest.reshape(1,1000)

And then I got:
Unknown array type passed to evaluation: labels array rank 1 with shape Rank: 1, DataType: DOUBLE, Offset: 0, Order: c, Shape: [1000], Stride: [1]. Labels and predictions must always be rank 2 or higher, with leading dimension being minibatch dimension

And I played around a bit and googled examples that work and somehow concluded I needed the stride, don’t remember.

Okay, well if I use .reshape(1000,1) I get the same error as above.
So instead I try:

Yhat.setShapeAndStride(new int[]{1000,1}, new int[]{1,1000});
Ytest.setShapeAndStride(new int[]{1000,1}, new int[]{1,1000});

But then when I do eval.eval(Ytest, Yhat); I get:

occurrences cannot be negative: -2778

You need to one-hot encode it.

I know. I was asking how. I figured surely dl4j has got a Yhat.onehotEncode() or something.

I guess I can implement the confusion matrix myself. I kind of figured that surely the Evaluation class has loads more features you wouldn’t want to be without. Anything worth keeping in it?

4

.reshape returns a new result. So if you just ran it like you posted here, it will do nothing at all.

        final INDArray y = Nd4j.create(1000);
        final INDArray yHat = Nd4j.create(1000);

        final Evaluation evaluation = new Evaluation();
        evaluation.eval(y.reshape(1000,1), yHat.reshape(1000,1));

        System.out.println(evaluation.toString());

This however will run just fine.

Looks like it isn’t mapped to a convenient method for direct execution, but it does exist: https://github.com/eclipse/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/api/ops/impl/shape/OneHot.java

This produces data that has the same shape as yours, and directly turns it into the appropriate shape for evaluation.

        final int numClasses = 15;
        final INDArray y = Nd4j.rand(1000).muli(numClasses).castTo(DataType.INT8).castTo(DataType.DOUBLE);
        final INDArray yHat = Nd4j.rand(1000).muli(numClasses).castTo(DataType.INT8).castTo(DataType.DOUBLE);

        final INDArray yOneHot = Nd4j.exec(new OneHot(y, null, numClasses, 1, 1.0, 0.0))[0];
        final INDArray yHatOneHot = Nd4j.exec(new OneHot(yHat, null, numClasses, 1, 1.0, 0.0))[0];

        final Evaluation evaluation = new Evaluation();
        evaluation.eval(yOneHot, yHatOneHot);

        System.out.println(evaluation);

If your data comes from outside of DL4J, and you get it as a list or array of integers, you can also just iterate over it for evaluation:

        int[] y = {3, 4, 14, 0};
        int[] yHat = {2, 4, 14, 1};

        final Evaluation evaluation = new Evaluation(15);
        for (int i = 0; i < y.length; i++) {
            evaluation.eval(yHat[i], y[i]);
        }

        System.out.println(evaluation);

So you don’t need to write your own after all :slight_smile:

1 Like