Saved network training update failure

DL4J
1

When I try to update a saved network with a new input dataset, the operation fails with the following error:

Invalid mask array: per-example masking should be a column vector, per output masking arrays should be the same shape as the output/labels arrays. Mask shape: [48, 33], output shape: [48, 2](layer name: layer8, layer index: 8, layer type: OutputLayer)

If I run the code to create a new neural network instead of starting from a restored network using the same dataset, the new network is created successfully.

Below are the summaries of net3, the network created from scratch, and the restored network.

What should I be looking at in order to resolve this problem? Is there any additional information I should provide?

Thanks

  • restored.summary() -
    ===========================================================================================================
    LayerName (LayerType) nIn,nOut TotalParams ParamsShape
    ===========================================================================================================
    layer0 (LSTM) 6,96 39,552 W:{6,384}, RW:{96,384}, b:{384}
    layer1 (SameDiffLayer) -,- 36,864 Wq:{2,48,96}, Wk:{2,48,96}, Wv:{2,48,96}, Wo:{96,96}
    layer2 (DenseLayer) 96,96 9,312 W:{96,96}, b:{96}
    layer3 (GlobalPoolingLayer) -,- 0 -
    layer4 (LSTM) 96,2 792 W:{96,8}, RW:{2,8}, b:{8}
    layer5 (SameDiffLayer) -,- 16 Wq:{2,1,2}, Wk:{2,1,2}, Wv:{2,1,2}, Wo:{2,2}
    layer6 (DenseLayer) 2,96 288 W:{2,96}, b:{96}
    layer7 (GlobalPoolingLayer) -,- 0 -
    layer8 (OutputLayer) 96,2 194 W:{96,2}, b:{2}
        Total Parameters:  87,018
    Trainable Parameters:  87,018
       Frozen Parameters:  0

===========================================================================================================

  • net3.summary() -
    ===========================================================================================================
    LayerName (LayerType) nIn,nOut TotalParams ParamsShape
    ===========================================================================================================
    layer0 (LSTM) 6,96 39,552 W:{6,384}, RW:{96,384}, b:{384}
    layer1 (SameDiffLayer) -,- 36,864 Wq:{2,48,96}, Wk:{2,48,96}, Wv:{2,48,96}, Wo:{96,96}
    layer2 (DenseLayer) 96,96 9,312 W:{96,96}, b:{96}
    layer3 (GlobalPoolingLayer) -,- 0 -
    layer4 (LSTM) 96,2 792 W:{96,8}, RW:{2,8}, b:{8}
    layer5 (SameDiffLayer) -,- 16 Wq:{2,1,2}, Wk:{2,1,2}, Wv:{2,1,2}, Wo:{2,2}
    layer6 (DenseLayer) 2,96 288 W:{2,96}, b:{96}
    layer7 (GlobalPoolingLayer) -,- 0 -
    layer8 (OutputLayer) 96,2 194 W:{96,2}, b:{2}
        Total Parameters:  87,018
    Trainable Parameters:  87,018
       Frozen Parameters:  0

===========================================================================================================

2

Problem solved by not processing train and test data with retrieved normalizer