LSTM Regression Example

DL4J
10

January 12 Update

I experimented with CSVVariableSlidingWindowRecordReader and was able to complete a training round (after some data format changes), but ultimately didn’t think it was the correct choice since it iterates up to a maximum window size and I expect a fixed window size of 12.

I have switched to using CSVSequenceRecordReader similar to the UCISequenceClassification example. I also changed my data format such that custom/train/features/0.csv contains the integers 0 to 11, each on a new line. The label associated with that sequence (12) is stored in custom/train/labels/0.csv. I hope this new data format works for regression problems, though I can’t say for sure yet. If you know better @treo, I’d welcome your feedback. I was hoping to have a single file with a moving window to avoid duplicating time series segments across multiple files (i.e. 0 to 11 => 12 and 1 to 12 => 13 both contain the segment 1 to 12), but I just couldn’t figure it out, so I’m falling back to this more verbose data format.

val baseDir = new File("src/main/resources/custom/")
val baseTrainDir = new File(baseDir, "train")
val featuresDirTrain = new File(baseTrainDir, "features")
val labelsDirTrain = new File(baseTrainDir, "labels")
val trainFeatures = new CSVSequenceRecordReader
trainFeatures.initialize(new NumberedFileInputSplit(featuresDirTrain.getAbsolutePath + "/%d.csv", 0, 1))
val trainLabels = new CSVSequenceRecordReader
trainLabels.initialize(new NumberedFileInputSplit(labelsDirTrain.getAbsolutePath + "/%d.csv", 0, 1))

val miniBatchSize = 1
val regression = true
val numPossibleLabels = -1
val labelIndex = 0
val dataSetIterator =
  new SequenceRecordReaderDataSetIterator(trainFeatures, trainLabels, miniBatchSize, numPossibleLabels, regression)

while (dataSetIterator.hasNext)
  println(dataSetIterator.next)
dataSetIterator.reset()

The dataSetIterator seems to have the expected contents:

===========INPUT===================
[[[         0,    1.0000,    2.0000,    3.0000,    4.0000,    5.0000,    6.0000,    7.0000,    8.0000,    9.0000,   10.0000,   11.0000]]]
=================OUTPUT==================
[[[12.0000]]]

The new exception I’m hunting down occurs during training:

val hiddenLayerWidth = 30
val net = new MultiLayerNetwork(
  new NeuralNetConfiguration.Builder()
    .optimizationAlgo(STOCHASTIC_GRADIENT_DESCENT)
    .seed(123)
    .weightInit(XAVIER)
    .list
    .layer(
      new LSTM.Builder()
        .activation(TANH)
        .nIn(1)
        .nOut(hiddenLayerWidth)
        .build)
    .layer(
      new RnnOutputLayer.Builder(MSE)
        .activation(IDENTITY)
        .nIn(hiddenLayerWidth)
        .nOut(1)
        .lossFunction(MSE)
        .build()
    )
    .build)
net.init()
logger.info("Starting training...")
net.setListeners(new ScoreIterationListener(1))
net.fit(dataSetIterator, 50)

The exception is:

Sequence lengths do not match for RnnOutputLayer input and labels:Arrays should be rank 3 with shape [minibatch, size, sequenceLength] - mismatch on dimension 2 (sequence length) - input=[1, 30, 12] vs. label=[1, 1, 1]

The error message is fairly clear, though I don’t know how to resolve it yet. Help would be welcomed. I’ll come back again tomorrow with another status update.