1 class was never predicted by the model and were excluded

MichalVavrikMuJ · May 10, 2020, 7:14am

Hello, I tried about 6 different NeuralNetConfiguration, but all of them resulted in the same issue - Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]. My latest configuration is kind of weird, as I was inspired in an existing code that I found here. For a while, I tought that problem was in my input, so I have already changed my batchsize from 1 to the size of the training set and my input are word2vec vector matrixes in DoublesDataSetIterator My input are word2vec vector matrixes in DoublesDataSetIterator, but nothing helped, after 10000 epochs, all the input is classified into a single class, totally ignoring the other one.

LayerName (LayerType) nIn,nOut TotalParams ParamsShape
layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}

layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}
Total Parameters: 541,502
Trainable Parameters: 541,502
Frozen Parameters: 0

2020-05-10 08:40:31.024 INFO 8392 — [nio-8200-exec-7] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 08:40:31.024 INFO 8392 — [nio-8200-exec-7] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2
Accuracy: 0.3395
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================

0 1
422 0 | 0 = 0
821 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

and I believe with my previous configuration I was doing better, yet the same problem occured:

+++++ Prediction Errors +++++
2020-05-09 20:50:11.877 INFO 9812 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2
Accuracy: 0.9038
Precision: 0.9038
Recall: 1.0000
F1 Score: 0.9494
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [0]

=========================Confusion Matrix=========================

0 1
0 448 | 0 = 0
0 4207 | 1 = 1
Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

. So basically what have I tried? I adjusted training set, batchsize, layers, weight init, activation functions. Does my problem lay in a neural network overfitting, or am I doing something fundamentally wrong? I am sorry to bother you, but I have run out of options known to me. Thank you, Michal

raver119 · May 10, 2020, 7:37am

Your training dataset is heavily biased towards 1 class I think

MichalVavrikMuJ · May 10, 2020, 1:05pm

I have adjusted training dataset on your advice and than retried with 2 different network config, both resulting in Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1] as you can see below:

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (OutputLayer) 600,2 1,202 W:{600,2}, b:{1,2}
        Total Parameters:  361,802
    Trainable Parameters:  361,802
       Frozen Parameters:  0
=======================================================================

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 15:00:05.598 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2

Accuracy: 0.5000
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================
0 1

405 0 | 0 = 0
405 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

and

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}
layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}
        Total Parameters:  541,502
    Trainable Parameters:  541,502
       Frozen Parameters:  0
=======================================================================

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

+++++ Prediction Errors +++++
2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

========================Evaluation Metrics========================

of classes: 2

Accuracy: 0.5000
Precision: 0.0000
Recall: 0.0000
F1 Score: 0.0000
Precision, recall & F1: reported for positive class (class 1 - “1”) only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]

=========================Confusion Matrix=========================
0 1

405 0 | 0 = 0
405 0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

I suppose there must be some bug in my network config, but I could use any hint where it could be found. Thank you

treo · May 11, 2020, 9:08am

There are a few things to note when defining and tuning your network.

Batch size is a hyper parameter.
a. If you put all of your data into a single batch, you will have to adjust other hyper parameters as well.
b. Increasing and reducing the batch size has effects on how often per epoch your network gets updated. You ideally want that each batch is just large enough to be a representative sample of your data.
How does the training loss change over time?
a. If it is flat, you may need to adjust your learning rate
b. if it goes down at first, but then starts going up, you are training for too long and should stop it earlier.
Are you entirely sure that you are producing valid input data?

Topic		Replies	Views
Simple CNN predicts NaNs DL4J	3	473	November 2, 2020
Layers is null for a MultiLayerNetwork object Tuning Help	0	250	February 3, 2022
Wrong input size - expected matrix DL4J	2	1273	June 1, 2020
Error: Labels and preOutput must have equal shapes DL4J	2	267	March 1, 2022
Methods for dropping out inputs (features) during post-fit Evaluation? DL4J	7	533	May 5, 2021

1 class was never predicted by the model and were excluded

layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}
Total Parameters: 541,502
Trainable Parameters: 541,502
Frozen Parameters: 0

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (OutputLayer) 600,2 1,202 W:{600,2}, b:{1,2}

of classes: 2

=========================Confusion Matrix=========================
0 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}
layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}

of classes: 2

=========================Confusion Matrix=========================
0 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

1 class was never predicted by the model and were excluded

layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2} Total Parameters: 541,502 Trainable Parameters: 541,502 Frozen Parameters: 0

2020-05-10 15:00:05.592 INFO 3168 — [nio-8200-exec-4] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600} layer1 (OutputLayer) 600,2 1,202 W:{600,2}, b:{1,2}

of classes: 2

=========================Confusion Matrix========================= 0 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

2020-05-10 14:29:25.293 INFO 8224 — [nio-8200-exec-3] m.v.d.services.NeuralNetworkService :

LayerName (LayerType) nIn,nOut TotalParams ParamsShape

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600} layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300} layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}

of classes: 2

=========================Confusion Matrix========================= 0 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times

Related Topics

layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}
Total Parameters: 541,502
Trainable Parameters: 541,502
Frozen Parameters: 0

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (OutputLayer) 600,2 1,202 W:{600,2}, b:{1,2}

=========================Confusion Matrix=========================
0 1

layer0 (DenseLayer) 600,600 360,600 W:{600,600}, b:{1,600}
layer1 (DenseLayer) 600,300 180,300 W:{600,300}, b:{1,300}
layer2 (OutputLayer) 300,2 602 W:{300,2}, b:{1,2}

=========================Confusion Matrix=========================
0 1