Incorrect regression output data

Hello! Please tell me what the problem might be.
I have a recurrent network for regression based on a sequence of 100 time steps.
An example of one step in the sequence:
[98.99920226267315, 98.74175067082459, 99.00645442019002, 98.69461164696499, 183762.0]
The network should predict the next time step, so I specify one step of the sequence as the label.
After applying the revert with the normalizer

INDArray timeSeriesOutput = net.output(finalData);
normalizer.revertFeatures(timeSeriesOutput);
normalizer.revertLabels(timeSeriesOutput);

I get various data like this:
[827.3333, 882.7206, 846.4208, 873.9750, 3.9037e6]
Which doesn’t even closely resemble the input of the 4 columns around the number 100.
Moreover, depending on the number of epochs, different numbers are always returned, sometimes exceeding the predicted values by several times. I have tried various configurations of the output layer, but the output data has never even been close to 100.
Can you suggest what I might be doing wrong?

@st331man yeah those numbers look weird. Can you setup a standalone example that’s easier to look at? That usually indicates that the internal distribution statistics are off somehow.

It also could mean your network wasn’t very accurate though.

@agibsonccc My code looks like this:

public class RegressionExperimental_ex {
    private static final String NEURAL_NAME = "data-d86660f4";
    private static final String DATASET_NAME = "/regression-datasets/" + NEURAL_NAME;
    public static File datasetName = new File(Configs.baseDir, DATASET_NAME + File.separator);
    private static File featuresDir = new File(datasetName, "features");
    private static File labelsDir = new File(datasetName, "labels");

    private static final int EPOCHS = 15;

    public static void main(String[] args) throws Exception {
        var files = Files.list(featuresDir.toPath()).collect(Collectors.toList());
        int filesSize = files.size();
        int eightyPrecents = (int) (filesSize * 0.80);

        //Train data
        SequenceRecordReader trainFeatures = new CSVSequenceRecordReader(0, ";");
        trainFeatures.initialize(new NumberedFileInputSplit(featuresDir.getAbsolutePath() + "/%d.csv", 1, eightyPrecents));
        SequenceRecordReader trainLabels = new CSVSequenceRecordReader(0, ";");
        trainLabels.initialize(new NumberedFileInputSplit(labelsDir.getAbsolutePath() + "/%d.csv", 1, eightyPrecents)); //17999

        boolean regression = true;
        int miniBatchSize = 30;
        int numLabelClasses = 1;
        SequenceRecordReaderDataSetIterator trainData = new SequenceRecordReaderDataSetIterator(trainFeatures, trainLabels, miniBatchSize, numLabelClasses,
                regression, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);

        //Normalize data
        NormalizerStandardize normalizer = new NormalizerStandardize();
        normalizer.fit(trainData);
        trainData.setPreProcessor(normalizer);
        trainData.reset();

        SequenceRecordReader testFeatures = new CSVSequenceRecordReader(0, ";");
        testFeatures.initialize(new NumberedFileInputSplit(featuresDir.getAbsolutePath() + "/%d.csv", eightyPrecents + 1, filesSize - 1)); //18000 - 21640
        SequenceRecordReader testLabels = new CSVSequenceRecordReader(0, ";");
        testLabels.initialize(new NumberedFileInputSplit(labelsDir.getAbsolutePath() + "/%d.csv", eightyPrecents + 1, filesSize - 1));

        SequenceRecordReaderDataSetIterator testData = new SequenceRecordReaderDataSetIterator(testFeatures, testLabels, miniBatchSize, numLabelClasses,
                regression, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);
        testData.setPreProcessor(normalizer); 

        // -- Final eval --
        SequenceRecordReader finalFeatures = new CSVSequenceRecordReader(0, ";");
        finalFeatures.initialize(new NumberedFileInputSplit(featuresDir.getAbsolutePath() + "/%d.csv", filesSize, filesSize));
        SequenceRecordReader finalLabels = new CSVSequenceRecordReader(0, ";");
        finalLabels.initialize(new NumberedFileInputSplit(labelsDir.getAbsolutePath() + "/%d.csv", filesSize, filesSize));

        SequenceRecordReaderDataSetIterator finalData = new SequenceRecordReaderDataSetIterator(finalFeatures, finalLabels, miniBatchSize, numLabelClasses,
                regression, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);
        finalData.setPreProcessor(normalizer);

        MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
                .seed(123)
                .weightInit(WeightInit.XAVIER)
                .updater(new Nadam())
                .list()
                .layer(new LSTM.Builder()
                        .activation(Activation.TANH)
                        .nIn(5)
                        .nOut(30)
                        .build())
                .layer(new LSTM.Builder()
                        .activation(Activation.TANH)
                        .nIn(30)
                        .nOut(14)
                        .build())
                .layer(new RnnOutputLayer.Builder(LossFunctions.LossFunction.MSE)
                        .activation(Activation.IDENTITY)
                        .nIn(30)
                        .nOut(5)
                        .build())
                .build();


        MultiLayerNetwork net = new MultiLayerNetwork(conf);
        net.init();
        System.out.printf("Started: miniBatchSize: %s, epohs: %s %n", miniBatchSize, EPOCHS);
        for (int i = 0; i < EPOCHS; i++) {
            long time = System.nanoTime();
            net.fit(trainData);
            var eval = net.evaluateRegression(testData);
            System.out.println(eval);
            System.out.println(eval.stats());

            time = System.nanoTime() - time;
            System.out.println(DATASET_NAME + " Time for epoch № " + (i + 1) + ": " + SECONDS.convert(time, TimeUnit.NANOSECONDS) + " sec");
        }
        System.out.println("Final:");
        RegressionEvaluation regEval = net.evaluateRegression(trainData);
        System.out.println(regEval);
        System.out.println("---------->");
        INDArray timeSeriesOutput = net.output(finalData);
        normalizer.revertFeatures(timeSeriesOutput);
        normalizer.revertLabels(timeSeriesOutput);
        System.out.println(timeSeriesOutput);
        long timeSeriesLength = timeSeriesOutput.size(2);
        INDArray lastTimeStepProbabilities = timeSeriesOutput.get(NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.point(timeSeriesLength - 1));
        System.out.println("Result: " + lastTimeStepProbabilities);
        double[] results = lastTimeStepProbabilities.toDoubleVector();
        System.out.println("Double arr: " + Arrays.toString(results));
        System.out.println("== End ==");
    }

And my dataset: data-d86660f4.zip - Google Drive

@st331man sorry pleas e ping me if I don’t reply. I get busy.

Looking at this again, ensure that you set preprocessing the labels to true. Label normalization is important.

Add this:
myNormalizer.fitLabel(true);

We don’t do that by default due to classifiers not normally needing that.

@agibsonccc Thank you for the hint; I didn’t know that I had to do it this way. Nevertheless, the problem turned out to be something else.
Apparently, the issue lies in the fact that I am incorrectly inputting data into the neural network, or I am doing it with an incorrect number of dimensions.
Each sequence is in a separate file, and each label is also in a separate file. Currently, I am using only one time step in the label, but in the future, I want to enable forecasting for several steps ahead.
It seems that I am somehow incorrectly feeding the data into the iterator, resulting in all the data mixing together, and ultimately yielding incorrect results. I noticed that the output somehow ends up containing identical data. For some reason, the network also outputs a sequence that contains repeated data, while I only need the label.

image