1 class was never predicted by the model and were excluded

Hello, I tried about 6 different NeuralNetConfiguration, but all of them resulted in the same issue - Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]. My latest configuration is kind of weird, as I was inspired in an existing code that I found here. For a while, I tought that problem was in my input, so I have already changed my batchsize from 1 to the size of the training set and my input are word2vec vector matrixes in DoublesDataSetIterator My input are word2vec vector matrixes in DoublesDataSetIterator, but nothing helped, after 10000 epochs, all the input is classified into a single class, totally ignoring the other one.

LayerName (LayerType) nIn,nOut TotalParams ParamsShape
layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}

layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}
Total Parameters: 541,502
Trainable Parameters: 541,502
Frozen Parameters: 0

2020-05-10 08:40:31.024 INFO 8392 — [nio-8200-exec-7] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 08:40:31.024 INFO 8392 — [nio-8200-exec-7] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2
Accuracy: 0.3395
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================

0 1
422 0 | 0 = 0
821 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

and I believe with my previous configuration I was doing better, yet the same problem occured:

+++++ Prediction Errors +++++
2020-05-09 20:50:11.877 INFO 9812 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2
Accuracy: 0.9038
Precision: 0.9038
Recall: 1.0000
F1 Score: 0.9494
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [0]

=========================Confusion Matrix=========================

0 1
0 448 | 0 = 0
0 4207 | 1 = 1
Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

. So basically what have I tried? I adjusted training set, batchsize, layers, weight init, activation functions. Does my problem lay in a neural network overfitting, or am I doing something fundamentally wrong? I am sorry to bother you, but I have run out of options known to me. Thank you, Michal

Your training dataset is heavily biased towards 1 class I think :slight_smile:

I have adjusted training dataset on your advice and than retried with 2 different network config, both resulting in Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]
as you can see below:

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (OutputLayer) 600,2 1,202 W:{600,2}, b:{1,2}

        Total Parameters:  361,802
    Trainable Parameters:  361,802
       Frozen Parameters:  0

=======================================================================

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 15:00:05.598 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2

Accuracy: 0.5000
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================
0 1

405 0 | 0 = 0
405 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

and

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}
layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}

        Total Parameters:  541,502
    Trainable Parameters:  541,502
       Frozen Parameters:  0

=======================================================================

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2

Accuracy: 0.5000
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================
0 1

405 0 | 0 = 0
405 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

I suppose there must be some bug in my network config, but I could use any hint where it could be found. Thank you

There are a few things to note when defining and tuning your network.

  1. Batch size is a hyper parameter.
    a. If you put all of your data into a single batch, you will have to adjust other hyper parameters as well.
    b. Increasing and reducing the batch size has effects on how often per epoch your network gets updated. You ideally want that each batch is just large enough to be a representative sample of your data.

  2. How does the training loss change over time?
    a. If it is flat, you may need to adjust your learning rate
    b. if it goes down at first, but then starts going up, you are training for too long and should stop it earlier.

  3. Are you entirely sure that you are producing valid input data?