Multi Label Sequence Classification

Tuning Help
1

Hello Everybody,
I am trying to do a multi label classification for sequences.
I started with a simple LSTM and used this as configuration:

    new NeuralNetConfiguration.Builder()
      .seed(seed)
      .updater(new Adam())
      .l2(1e-5)
      .weightInit(WeightInit.XAVIER)
      .biasInit(0)
      .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue)
      .gradientNormalizationThreshold(10.0)
      .trainingWorkspaceMode(WorkspaceMode.ENABLED)
      .inferenceWorkspaceMode(WorkspaceMode.ENABLED)
      .list()
      .layer(
        0,
        new LSTM.Builder()
          .nIn(numberOfFeatures)
          .nOut(100)
          .activation(Activation.TANH)
          .build()
      )
      .layer(
        1,
        new RnnOutputLayer.Builder()
          .activation(Activation.SIGMOID)
          .weightInit(WeightInit.XAVIER)
          .lossFunction(new LossMultiLabel())
          .nIn(100)
          .nOut(numberOfLabels)
          .build()
      )
      .build()

but after the training the evaluation of the network always looks like this:
grafik

To evaluate this i used a RocBinary.
Is there something obviously wrong with what I am doing here?
If I use LossBinaryXENT instead of LossMultiLabel, the same thing happens.
I hope someone can help me with this.
Greetings Quack

2

To give a little bit more information, here are my train and test iterators:

    val trainFeatures = new CSVSequenceRecordReader();
    trainFeatures.initialize(
      new NumberedFileInputSplit(
        trainFolder + "/%d.csv",
        0,
        numberOfTrainFiles - 1
      )
    );

    val trainLabels = new CSVSequenceRecordReader();
    trainLabels.initialize(
      new NumberedFileInputSplit(
        trainLabelFolder + "/%d.csv",
        0,
        numberOfTrainFiles - 1
      )
    );

    val trainIter = new SequenceRecordReaderDataSetIterator(
      trainFeatures,
      trainLabels,
      batchSize,
      numberOfLabels,
      false,
      SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END
    );

    val testFeatures = new CSVSequenceRecordReader()
    testFeatures.initialize(
      new NumberedFileInputSplit(
        testFolder + "/%d.csv",
        trainMinIdx,
        trainMaxIdx
      )
    )

    val testLabels = new CSVSequenceRecordReader()
    testLabels.initialize(
      new NumberedFileInputSplit(
        testLabelFolder + "/%d.csv",
        testMinIdx,
        testMaxIdx
      )
    )

    val testIter = new SequenceRecordReaderDataSetIterator(
      testFeatures,
      testLabels,
      batchSize,
      numberOfLabels,
      false,
      SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END
    )

I realy need help with this issue, there dont seem to be any examples for multi label classification, and i dont know if i am doing something wrong here.
Greetings Duck

3

have you tried a practice project where you use a lstm net to predict a sine wave first?

4

Yes, I have build LSTM networks for classification and regression before, i just cannot seem to make the multi label classification work.

5

I am not sure how exactly multi-label works. but have you trid some other activation function for the output layer like SOFTMAX which is often used for classification problems?