Sequential model tuning suggestions

Hello, I’m making a model for music generation and looking for suggestions on how to improve it. The dataset consists of a feature and label file, respectively.

A single feature is a 32 timestep long sequence, and each timestep is a 24-d vector with elements -1, 0 or 1.

A label is a single timestep, identical as those from the features.
In CSV a feature/label pair looks as such:

...  29 more rows


This is my current model setup:

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
                .updater(new Adam())
                .layer(new LSTM.Builder().activation(Activation.RELU6).nIn(24).nOut(100).build())
                .layer(new RnnOutputLayer.Builder(LossFunctions.LossFunction.MEAN_ABSOLUTE_ERROR)

While generating new music, the idea is to convert output double values to integers, using the nearest neighbour method (or something similar) and then feed those to the network as a next step.

To add a bit more context, this how i prepare the data

FileSplit featureIS = new FileSplit(FEAT_PATH);
        SequenceRecordReader trainFeats = new CSVNLinesSequenceRecordReader(32);

        FileSplit labelsIS = new FileSplit(LABEL_PATH);
        SequenceRecordReader trainLabels = new CSVNLinesSequenceRecordReader(1);

DataSetIterator train = new SequenceRecordReaderDataSetIterator(trainFeats, trainLabels, 10,
                24, true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);