Sequential model tuning suggestions

Tuning Help
1

Hello, I’m making a model for music generation and looking for suggestions on how to improve it. The dataset consists of a feature and label file, respectively.

A single feature is a 32 timestep long sequence, and each timestep is a 24-d vector with elements -1, 0 or 1.

A label is a single timestep, identical as those from the features.
In CSV a feature/label pair looks as such:

--feature--
0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0
0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,-1,0,0,1,0,0,0,1,0
...
...  29 more rows
...
0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0

--label--
0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0

This is my current model setup:

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
                .weightInit(WeightInit.XAVIER)
                .updater(new Adam())
                .gradientNormalizationThreshold(1)
                .list()
                .layer(new LSTM.Builder().activation(Activation.RELU6).nIn(24).nOut(100).build())
                .layer(new RnnOutputLayer.Builder(LossFunctions.LossFunction.MEAN_ABSOLUTE_ERROR)
                        .activation(Activation.TANH).nIn(100).nOut(24).build())
                .build();
2

While generating new music, the idea is to convert output double values to integers, using the nearest neighbour method (or something similar) and then feed those to the network as a next step.

To add a bit more context, this how i prepare the data

FileSplit featureIS = new FileSplit(FEAT_PATH);
        SequenceRecordReader trainFeats = new CSVNLinesSequenceRecordReader(32);
        trainFeats.initialize(featureIS);

        FileSplit labelsIS = new FileSplit(LABEL_PATH);
        SequenceRecordReader trainLabels = new CSVNLinesSequenceRecordReader(1);
        trainLabels.initialize(labelsIS);

DataSetIterator train = new SequenceRecordReaderDataSetIterator(trainFeats, trainLabels, 10,
                24, true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);