NullPointerException org.nd4j.autodiff.samediff.internal.Variable.getOutputOfOp()

SameDiff
141

@agibsonccc I appreciate your pointers regarding debugging using an ide. I have been using both Eclipse and IntelliJ for more than a decade. And I appreciate all the work you have done, and your patience. I am glad that the flabuffers change I prompted you to make is benefitting your project.

I removed ALL code related to evaluation. As expected, the exact same failure occurs. As I sadi, I will work on this on my own and hoefully I will not have to bother you again about it.

Thnk you for all your help. Have a good day

142

@adonnini ok good. So let’s troubleshoot the next part. I showed you the error from the normalizer there and the code snippet.

Something from the normalizer is causing the means to be null. The normalizer tracks running means in order to properly normalize and restore values.
What I asked you to do was look in to the cause of that. I gave you the way that I would go about doing that.

Without throwing that error, it would just throw a null pointer. Try to track down the code where you are using normalizers. That can happen manually, or as part of an iterator. That can also be used when you pass in data.

143

@agibsonccc

I set the debugger breakpoint at the line where execution fails. Stepping into the code doesn’t get me anywhere as I do not know your code (I tried). I am attaching screenshots of the debug window. Please let me know what other information you need from the debugger. The content of the console is a big text file which exceeds the size allowed here. I can send you selections if you want. I am not sure what you are looking for exactly.

I use normalizer (NormalizerStandardize) only when preparing the data for running the model (see code below). Whether I also use normalizers as part of an iterator, I have no idea. How would I find out without knowing more about your code?

CODE WHERE I USE NORMALIZER EXPLICITLY

       normalizer = new NormalizerStandardize();
        normalizer.fitLabel(true);
        normalizer.fit(trainData);           //Collect the statistics (mean/stdev) from the training data. This does not modify the input data

        trainData.reset();
        testData.reset();

        while(trainData.hasNext()) {
            normalizer.transform(trainData.next());     //Apply normalization to the training data
        }

        while(testData.hasNext()) {
            normalizer.transform(testData.next());         //Apply normalization to the test data. This is using statistics calculated from the *training* set
        }

        trainData.reset();
        testData.reset();

        trainData.setPreProcessor(normalizer);
        testData.setPreProcessor(normalizer);

Screenshot from 2025-09-30 10-20-51
Screenshot from 2025-09-30 10-20-513819×1901 340 KB

Screenshot from 2025-09-30 10-14-17
Screenshot from 2025-09-30 10-14-173819×1647 389 KB

144

@adonnini that’s ok I just needed what tyou’re seeing. This was super helpful thank you!

So let’s break this down. You’re using NormalizeStandardize.

This uses a standardize strategy with DistributionStats which seem to have null values:

Your mean in the stats here appears to be null.

You’re using the csv sequence record reader which means you’re using the RecordReaderDataSetIterator.

That iterator on next() at some point either returns null:

     while(trainData.hasNext()) {             normalizer.transform(trainData.next());     *//Apply normalization to the training data*         }          while(testData.hasNext()) {             normalizer.transform(testData.next());         *//Apply normalization to the test data. This is using statistics calculated from the training set         }

OR you’re wiring up the normalize standardize wrong.
Fit SHOULD* be setting the mean and standard deviation of your data.

My first question is when is this null? Is it after loading the data data from disk?

We might either have a restored serializer bug or some silent failure in the iterator which then causes the null mean and standard deviation.

What I would do next is set a conditional break point on when the mean or standard deviation are null.

For a reminder your stack trace was this:
Exception in thread “main” java.lang.RuntimeException: No data was added, statistics cannot be determined
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:161)
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:85)
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:112)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.prepareTrainingAndTestData(LocationNextNeuralNetworkV7_04.java:2163)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.sameDiff3(LocationNextNeuralNetworkV7_04.java:228)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.main(LocationNextNeuralNetworkV7_04.java:221)

So you prepare the training and test data, you call fit and for whatever reason that’s null.
You’re using an iterator so some aspect of that returns null.

One work around might be to instead of using the iterator, loop through it yourself with while(data.hasNext()) and pass that to fit. You can see how the iterator code works by stepping in to it and just copying that.

If you want, you could also use the conditional breakpoint checking if next() returns null at any point.

That would be the main thing I would do first. Depending on whether prepare is from a saved model or from scratch I would try to figure out what’s causing this.

145

@agibsonccc

I tried to loop through traindata manually, execution failed in the same way while trying to process the first train dataset.

I set a breakpoint inside the loop. below is the screenshot.

Screenshot from 2025-10-03 09-19-55
Screenshot from 2025-10-03 09-19-553819×2038 508 KB

What did you mean when you said

I did not change this section of the code. It’s always been as it is now. I have several models where it runs without problems.

Thanks

146

@agibsonccc

I set a breakpoint at normalizer.fit in a model where execution does not fail. Please find the debug screenshot below.

Screenshot from 2025-10-03 09-22-42
Screenshot from 2025-10-03 09-22-423819×1487 353 KB

The main difference, that I can see, from the same screenshot from the model where execution fails is in the value of ``underlying`` which in this screenshot is null.

What do you think?

147

@adonnini this is great! I can definitely work like this. From what I can see your featureStats and the like are still null right there. We really need to figure out why. I can see you have the 2 record readers. I am still wondering if there’s a silent failure of some kind there. I would set a conditional breakpoint on any of the record readers returning null.

You can search for a class on your classpath and download the source as well as use breakpoints on them. You can also control click in to the class. Find where the reader returns next() and look for nul anywhere and see what that might be.

148

@agibsonccc Thanks for your encouragement. Unfortunately, it does not get me any closer to resolving this issue.

Also unfortunately, I have several higher priority tasks to work on.

I literally do not have the time to leanr how your code works. I am going to have to put work on this on-hold for the time being.

Thanks for your help and encoragement.

149

@adonnini actually I’m not really asking you to. There’s also not much to it: the iterator has a next(). The iterator calls the record reader().

The conditional breakpoint being null would tell you if anything is happening or not. I’m not really asking you to understand anything I don’t think?

I gave you the logic, where to put it and what to look for. That’s really all I’d expect, not really for you to chase everything all over the code base. It’s up to you if you want to take the steps I’m telling you to or not. Ping me again if you decide to and I’m happy to take a further look or clarirfy things.

Broadly, debuggers are standard tools you use in every day life to just understand what something looks like. You were doing exactly that. Sometimes it helps you understand if you’re using something correctly or where an issue is occurring and why.

What WOULD be a lot more work would be me telling you to reverse engineer the full code base. I’m incrementally just trying to discover where the source of your issue is and doing the steps that I would normally do myself to troubleshoot such an issue.

Data pipelines CAN have bugs and silent failures. The statistics being null is unusual and needs to be diagnosed. If your input data is wrong and there’s a silent failure that should be fixe.d

150

I stepped through the code whne I set the breakpoint at

normalizer.fit(trainData.next());

Below you will find the path that leads to the

No data was added, statistics cannot be determined

exception.

Based on my untrained reading, It looks like the dataset used to create featureStats and labelStats is non-empty.

I don’t see then where runningMean used of the test in DistributionStats is produced.

What of you make of it?

Thanks

My code
-------
            normalizer.fit(trainData.next());     //Apply normalization to the training data

-------------------------------------------------------------------------------------------------------------------------------------------------

ABSTRACTDATASETNORMALIZER
=========================
    public void fit(DataSet dataSet) {
        this.featureStats = this.newBuilder().addFeatures(dataSet).build();
        if (this.isFitLabel()) {
            this.labelStats = this.newBuilder().addLabels(dataSet).build();
        }

dataSet = {DataSet@2680} "===========INPUT===================\n[[[   45.7535,   45.7535,         0,  ...         0,         0,         0], \n[    8.3126,    8.3126,         0,  ...         0,         0,         0], \n[ 1.6862e12, 1.6862e12,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[    1.0000,    1.0000,         0,  ...         0,         0,         0]], \n\n[[   45.7534,   45.7534,         0,  ...         0,         0,         0], \n[    8.3122,    8.3122,         0,  ...         0,         0,         0], \n[ 1.6862e12, 1.6862e12,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[    1.0000,    1.0000,         0,  ...         0,         0,         0]], \n\n[[   45.7535,         0,         0,  ...         0,      "
 columnNames = {ArrayList@2683}  size = 0
 labelNames = {ArrayList@2684}  size = 0
 features = {NDArray@2685} "[[[   45.7535,   45.7535,         0,  ...         0,         0,         0], \n[    8.3126,    8.3126,         0,  ...         0,         0,         0], \n[ 1.6862e12, 1.6862e12,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[    1.0000,    1.0000,         0,  ...         0,         0,         0]], \n\n[[   45.7534,   45.7534,         0,  ...         0,         0,         0], \n[    8.3122,    8.3122,         0,  ...         0,         0,         0], \n[ 1.6862e12, 1.6862e12,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[    1.0000,    1.0000,         0,  ...         0,         0,         0]], \n\n[[   45.7535,         0,         0,  ...         0,         0,         0], \n[    8.3126,     "
 labels = {NDArray@2686} "[[[         0,   45.7535,         0,  ...         0,         0,         0], \n[         0,    8.3127,         0,  ...         0,         0,         0]], \n\n[[         0,   45.7533,         0,  ...         0,         0,         0], \n[         0,    8.3124,         0,  ...         0,         0,         0]], \n\n[[   45.7535,         0,         0,  ...         0,         0,         0], \n[    8.3126,         0,         0,  ...         0,         0,         0]], \n\n ..., \n\n[[         0,   45.7535,         0,  ...         0,         0,         0], \n[         0,    8.3120,         0,  ...         0,         0,         0]], \n\n[[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0]], \n\n[[   45.7534,         0,         0,  ...         0,         0,         0], \n[    8.3122,         0,         0,  ...         0,         0,         0]]]"
 featuresMask = {NDArray@2687} "[[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n ..., \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0]]"
 labelsMask = {NDArray@2688} "[[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n ..., \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0], \n[         0,         0,         0,  ...         0,         0,         0]]"
 exampleMetaData = null
 preProcessed = false
 
 -------------------------------------------------------------------------------------------------------------------------------------------------
 
 DISTRIBUTIONSTATS
 ==================
 
        public DistributionStats build() {
            if (this.runningMean == null) {
                throw new RuntimeException("No data was added, statistics cannot be determined");
            } else {
                return new DistributionStats(this.runningMean.dup(), Transforms.sqrt(this.runningVariance, true));
            }
        }
    }
}

EXCEPTION "No data was added, statistics cannot be determined" THROWN
151

@adonnini thanks for being willing to take a closer look at this. I’ll look at this and interpret the output if you keep giving me information. The conditional breakpoint I asked you to set should stop if any of them return null for any reason. Did you try that yet?

For your current question, the data set looks fine. A value SHOULD be being set if you pass fit in.
Can you paste the call trace in so I can see it fully? I can both diff that against SNAPSHOTS and 1.0.0-M2.1.

152

@agibsonccc Below you will find the traceback

I sent you the sequence of steps when stepping into the code at the conditional breakpoint. What am I not understanding? To be clear I set the breakpoint at

 normalizer.fit(trainData.next());

Where else should I put it?

Execution stops/fails when runningMean returns a value of null

I thought I labeled all the steps clearly.

The mystery, to me is where ``runningMean``` gets set as the execution fails when build() tests for the value of runningMean

TRACEBACK

==========

Exception in thread "main" java.lang.RuntimeException: No data was added, statistics cannot be determined
	at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:161)
	at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:85)
	at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:75)
	at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.sameDiff3(LocationNextNeuralNetworkV7_04.java:295)
	at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.main(LocationNextNeuralNetworkV7_04.java:221)
153

sorry needed time to look at this.

So what I’m seeing here:

@Override
public void fit(DataSet dataSet) {
    featureStats = (S) newBuilder().addFeatures(dataSet).build();
    if (isFitLabel()) {
        labelStats = (S) newBuilder().addLabels(dataSet).build();
    }
}

This passes in a dataset. The data set would have to be null here.

You do add features:

Builder addFeatures(org.nd4j.linalg.dataset.api.DataSet dataSet);

DistributionFeatures implementation is:

/**
 * Add the features of a DataSet to the statistics
 */
public Builder addFeatures(@NonNull org.nd4j.linalg.dataset.api.DataSet dataSet) {
    return add(dataSet.getFeatures(), dataSet.getFeaturesMaskArray());
}

add is this:

public Builder add(@NonNull INDArray data, INDArray mask) {
    data = DataSetUtil.tailor2d(data, mask);
    
    // Using https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
    if (data == null) {
        // Nothing to add. Either data is empty or completely masked. Just skip it, otherwise we will get
        // null pointer exceptions.
        return this;
    }
    
    INDArray mean = data.mean(0).reshape(1, data.size(1));
    INDArray variance = data.var(false, 0).reshape(1, data.size(1));
    long count = data.size(0);
    
    if (runningMean == null) {
        // First batch
        runningMean = mean;
        runningVariance = variance;
        runningCount = count;
        if (data.size(0) == 1) {
            //Handle edge case: currently, reduction ops may return the same array
            //But we don't want to modify this array in-place later
            runningMean = runningMean.dup();
            runningVariance = runningVariance.dup();
        }
    } else {
        // Update running variance
        INDArray deltaSquared = Transforms.pow(mean.subRowVector(runningMean), 2);
        INDArray mB = variance.muli(count);
        runningVariance.muli(runningCount).addiRowVector(mB)
                        .addiRowVector(deltaSquared
                                        .muli((float) (runningCount * count) / (runningCount + count)))
                        .divi(runningCount + count);
        // Update running count
        runningCount += count;
        // Update running mean
        INDArray xMinusMean = data.subRowVector(runningMean);
        runningMean.addi(xMinusMean.sum(0).divi(runningCount));
    }
    
    return this;
}

Your runningMean being null means that one of these code paths happened:

data null

so here:

if (data == null) {
    // Nothing to add. Either data is empty or completely masked. Just skip it, otherwise we will get
    // null pointer exceptions.
    return this;
}

When running mean is null the value is set for the first time. So this can happen when the first data set is null.

build() is called():

public DistributionStats build() {
    if (runningMean == null) {
        throw new RuntimeException("No data was added, statistics cannot be determined");
    }
    return new DistributionStats(runningMean.dup(), Transforms.sqrt(runningVariance, true));
}

@adonnini apologies formatting. I cleaned that up. What I was showing you there is the code block there does run in to that. I agree that it’s odd.

So let’s do this. Let’s confirm this is a regression of some kind. Can you try just your data set loading code with M2.1 vs snapshots and see if anything in just THAT business logic changed?
My understanding is that before snapshots literally all of your code worked fine. All I want to do is continue narrowing this down. If not I’ll have to look at your case a bit more. There’s literally no reason that that should be happening regardless though.

I can see in at least what you’re showing me that yes: there’s no null dataset. Regardless, that code path is happening for some reason.

154

@agibsonccc I don’t quite understand your comment. A large part of it is all crossed out.

In any event, as evidenced by the dub output (see below) the dataset I pass to fit is NOT null. The information below comes straight from the deugger. I did not touch it.

What am I missing?

My code

        normalizer.fit(trainData.next());     //Apply normalization to the training data

ABSTRACTDATASETNORMALIZER

public void fit(DataSet dataSet) {
    this.featureStats = this.newBuilder().addFeatures(dataSet).build();
    if (this.isFitLabel()) {
        this.labelStats = this.newBuilder().addLabels(dataSet).build();
    }

dataSet = {DataSet@2680} "===========INPUT===================\n[[[ 45.7535, 45.7535, 0, … 0, 0, 0], \n[ 8.3126, 8.3126, 0, … 0, 0, 0], \n[ 1.6862e12, 1.6862e12, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 1.0000, 1.0000, 0, … 0, 0, 0]], \n\n[[ 45.7534, 45.7534, 0, … 0, 0, 0], \n[ 8.3122, 8.3122, 0, … 0, 0, 0], \n[ 1.6862e12, 1.6862e12, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 1.0000, 1.0000, 0, … 0, 0, 0]], \n\n[[ 45.7535, 0, 0, … 0, "
columnNames = {ArrayList@2683} size = 0
labelNames = {ArrayList@2684} size = 0
features = {NDArray@2685} "[[[ 45.7535, 45.7535, 0, … 0, 0, 0], \n[ 8.3126, 8.3126, 0, … 0, 0, 0], \n[ 1.6862e12, 1.6862e12, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 1.0000, 1.0000, 0, … 0, 0, 0]], \n\n[[ 45.7534, 45.7534, 0, … 0, 0, 0], \n[ 8.3122, 8.3122, 0, … 0, 0, 0], \n[ 1.6862e12, 1.6862e12, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 1.0000, 1.0000, 0, … 0, 0, 0]], \n\n[[ 45.7535, 0, 0, … 0, 0, 0], \n[ 8.3126, "
labels = {NDArray@2686} “[[[ 0, 45.7535, 0, … 0, 0, 0], \n[ 0, 8.3127, 0, … 0, 0, 0]], \n\n[[ 0, 45.7533, 0, … 0, 0, 0], \n[ 0, 8.3124, 0, … 0, 0, 0]], \n\n[[ 45.7535, 0, 0, … 0, 0, 0], \n[ 8.3126, 0, 0, … 0, 0, 0]], \n\n …, \n\n[[ 0, 45.7535, 0, … 0, 0, 0], \n[ 0, 8.3120, 0, … 0, 0, 0]], \n\n[[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0]], \n\n[[ 45.7534, 0, 0, … 0, 0, 0], \n[ 8.3122, 0, 0, … 0, 0, 0]]]”
featuresMask = {NDArray@2687} “[[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n …, \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0]]”
labelsMask = {NDArray@2688} “[[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n …, \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0], \n[ 0, 0, 0, … 0, 0, 0]]”
exampleMetaData = null
preProcessed = false

DISTRIBUTIONSTATS

    public DistributionStats build() {
        if (this.runningMean == null) {
            throw new RuntimeException("No data was added, statistics cannot be determined");
        } else {
            return new DistributionStats(this.runningMean.dup(), Transforms.sqrt(this.runningVariance, true));
        }
    }
}

}

EXCEPTION “No data was added, statistics cannot be determined” THROWN

155

@agibsonccc I ma sorry for going silent for a month. I thought I was wiating for your response. Sorry! I will proceed with trying the code with M2.1. I will let you know as soon as I have an update. Thanks

156

@agibsonccc I ran the application with M2.1. The result was exactly the same. I also ran the debugger. Exact same output.

Below you will find a complete copy of the POM file for you to verify that I enabled use of M2.1 correctly.

At this point, I really don’t know what to do next. One fact is that other models with the “same” train and test dataset code run without problems. I’ll take a closer look at that code.

Please let me know if you have any thoughts, questions, or you would like me to try something else.

Thanks

POM

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~
  ~ This program and the accompanying materials are made available under the
  ~  terms of the Apache License, Version 2.0 which is available at
  ~  https://www.apache.org/licenses/LICENSE-2.0.
See the NOTICE file distributed with this work for additional
information regarding copyright ownership.
  ~
  ~   See the NOTICE file distributed with this work for additional
  ~   information regarding copyright ownership.
  ~  Unless required by applicable law or agreed to in writing, software
  ~  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
  ~  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
  ~  License for the specific language governing permissions and limitations
  ~  under the License.
  ~
  ~  SPDX-License-Identifier: Apache-2.0
  ~
  ~
  -->

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
    <groupId>org.nd4j</groupId>
    <artifactId>TrajectoryPredictionTransformer</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <name>Introduction to DL4J</name>
    <description>A set of examples introducing the DL4J framework</description>

    <properties>
        <dl4j-master.version>1.0.0-M2.1</dl4j-master.version>
<!--        <dl4j-master.version>1.0.0-SNAPSHOT</dl4j-master.version>-->
        <!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
        <!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
        <nd4j.backend>nd4j-native</nd4j.backend>
        <java.version>1.8</java.version>
        <maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
        <maven.minimum.version>3.3.1</maven.minimum.version>
        <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
        <maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
        <jcommon.version>1.0.24</jcommon.version>
        <jfreechart.version>1.0.13</jfreechart.version>
        <logback.version>1.4.6</logback.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>5.9.2</junit.version>
        <javacv.version>1.5.7</javacv.version>

    </properties>


    <repositories>

        <repository>
            <id>central-portal-snapshots</id>
            <name>Central Portal Snapshots</name>
            <url>https://central.sonatype.com/repository/maven-snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>

        <repository>
            <id>sonatype-nexus-snapshots</id>
            <name>Sonatype Nexus Snapshots</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>

        <repository>
            <id>central</id>
            <name>Maven Central</name>
            <layout>default</layout>
            <url>https://repo1.maven.org/maven2</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

        <repository>
            <id>maven-google</id>
            <name>Maven.Google</name>
            <layout>default</layout>
            <url>https://maven.google.com/</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

    </repositories>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.google.flatbuffers</groupId>
                <artifactId>flatbuffers-java</artifactId>
                <version>25.2.10</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.backend}</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.datavec</groupId>
            <artifactId>datavec-api</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.datavec</groupId>
            <artifactId>datavec-data-image</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.datavec</groupId>
            <artifactId>datavec-local</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-datasets</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>resources</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-zoo</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- ParallelWrapper & ParallelInference live here -->
        <dependency>
            <!--            <groupId>org.eclipse.deeplearning4j</groupId>-->
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-parallel-wrapper</artifactId>
            <version>${dl4j-master.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.flatbuffers</groupId>
                    <artifactId>flatbuffers-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- Used in the feedforward/classification/MLP* and feedforward/regression/RegressionMathFunctions example -->
<!--        <dependency>-->
<!--            <groupId>jfree</groupId>-->
<!--            <artifactId>jfreechart</artifactId>-->
<!--            <version>${jfreechart.version}</version>-->
<!--        </dependency>-->
<!--        <dependency>-->
<!--            <groupId>org.jfree</groupId>-->
<!--            <artifactId>jcommon</artifactId>-->
<!--            <version>${jcommon.version}</version>-->
<!--        </dependency>-->

        <!-- Used for downloading data in some of the examples -->
<!--        <dependency>-->
<!--            <groupId>org.apache.httpcomponents</groupId>-->
<!--            <artifactId>httpclient</artifactId>-->
<!--            <version>4.5.14</version>-->
<!--        </dependency>-->

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>${logback.version}</version>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv-platform</artifactId>
            <version>${javacv.version}</version>
        </dependency>

        <!-- Test dependency. Ignore for your own application. -->
<!--        <dependency>-->
<!--            <groupId>org.junit.jupiter</groupId>-->
<!--            <artifactId>junit-jupiter-engine</artifactId>-->
<!--            <version>${junit.version}</version>-->
<!--            <scope>test</scope>-->
<!--        </dependency>-->

<!--        <dependency>-->
<!--            <groupId>org.junit.jupiter</groupId>-->
<!--            <artifactId>junit-jupiter-api</artifactId>-->
<!--            <version>${junit.version}</version>-->
<!--            <scope>test</scope>-->
<!--        </dependency>-->

<!--        <dependency>-->
<!--            <groupId>org.tensorflow</groupId>-->
<!--            <artifactId>tensorflow</artifactId>-->
<!--            <version>1.15.0</version>-->
<!--        </dependency>-->
<!--        <dependency>-->
<!--            <groupId>org.apache.arrow</groupId>-->
<!--            <artifactId>arrow-format</artifactId>-->
<!--            <version>6.0.1</version>-->
<!--        </dependency>-->

        <dependency>
            <groupId>com.google.flatbuffers</groupId>
            <artifactId>flatbuffers-java</artifactId>
            <version>25.2.10</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.opencsv/opencsv -->
        <dependency>
            <groupId>com.opencsv</groupId>
            <artifactId>opencsv</artifactId>
            <version>5.11.2</version>
        </dependency>

    </dependencies>

    <!-- Maven Enforcer: Ensures user has an up to date version of Maven before building -->
    <build>
        <plugins>

            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                <inherited>true</inherited>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit-platform</artifactId>
                        <version>3.0.0-M5</version>
                    </dependency>
                </dependencies>
            </plugin>

            <plugin>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>1.0.1</version>
                <executions>
                    <execution>
                        <id>enforce-default</id>
                        <goals>
                            <goal>enforce</goal>
                        </goals>
                        <configuration>
                            <rules>
                                <requireMavenVersion>
                                    <version>[${maven.minimum.version},)</version>
                                    <message>********** Minimum Maven Version is ${maven.minimum.version}. Please upgrade Maven before continuing (run "mvn --version" to check). **********</message>
                                </requireMavenVersion>
                            </rules>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven-compiler-plugin.version}</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>${exec-maven-plugin.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <executable>java</executable>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${maven-shade-plugin.version}</version>
                <configuration>
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                    <shadedClassifierName>${shadedClassifier}</shadedClassifierName>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>org/datanucleus/**</exclude>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>com.lewisd</groupId>
                                        <artifactId>lint-maven-plugin</artifactId>
                                        <versionRange>[0.0.11,)</versionRange>
                                        <goals>
                                            <goal>check</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore/>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>
157

@adonnini sorry for the delay. I still feel like that means that for some reason the pipeline is returning null on one of the datasets or there’s a serialization failure somewhere. I’d have to look at the record reader to see what could be causing that to return null. It’s definitely strange it’s doing that. Either way, if something fails it should fail loudly.

158

No problem! You have higher priorities.

I ran a test. I ran a model I created previously and the model I am working on now. They both have the code that sets up train and test data and the normalizer.

Execution of the older model runs to successful completion while the current model fails at ``normalizer.fit```

You will find the relevant code section from both models below, less than 100 lines. Please take a look and tell me if you can see any material difference which cause one to run and the other to fail.

Thanks

THIS CODE EXECUTES TO COMPLETION SUCCESSFULLY
================================================

        // ----- Load the training data -----
        SequenceRecordReader trainFeatures = new CSVSequenceRecordReader();
        trainFeatures.initialize(new NumberedFileInputSplit(featuresDirTrain.getAbsolutePath() + "/%d.csv", 0, lastTrainCount));
        SequenceRecordReader trainLabels = new CSVSequenceRecordReader();
        trainLabels.initialize(new NumberedFileInputSplit(labelsDirTrain.getAbsolutePath() + "/%d.csv", 0, lastTrainCount));

        DataSetIterator trainData2 = new SequenceRecordReaderDataSetIterator(trainFeatures, trainLabels, miniBatchSize, numLabelClasses,
                true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);

        // ----- Load the test data -----
        //Same process as for the training data.
        SequenceRecordReader testFeatures = new CSVSequenceRecordReader();
        testFeatures.initialize(new NumberedFileInputSplit(featuresDirTest.getAbsolutePath() + "/%d.csv", 0, lastTestCount));
        SequenceRecordReader testLabels = new CSVSequenceRecordReader();
        testLabels.initialize(new NumberedFileInputSplit(labelsDirTest.getAbsolutePath() + "/%d.csv", 0, lastTestCount));

        DataSetIterator testData2 = new SequenceRecordReaderDataSetIterator(testFeatures, testLabels, miniBatchSize, numLabelClasses,
                true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);

        System.out.println( TAG+" "+" Printing traindata dataset shape");
        DataSet data = trainData2.next();
        System.out.println(java.util.Arrays.toString(data.getFeatures().shape()));

        System.out.println( TAG+" "+" Printing testdata dataset shape");
        DataSet data2 = testData2.next();
        System.out.println(java.util.Arrays.toString(data2.getFeatures().shape()));

        INDArray predicted2 = null;

        //NETWORK CONFIGURATION SET-UP AND NETWORK INIT - START - [=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=][=]

        NormalizerStandardize normalizer2 = new NormalizerStandardize();

        normalizer2.fitLabel(true);
        normalizer2.fit(trainData2);           //Collect the statistics (mean/stdev) from the training data. This does not modify the input data

        trainData2.reset();

///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

THe CODE BELOW FAILS TO RUN TO COMPLETION FAILING AT THIS LINE
==========================================================
//            normalizer.fit(trainData.next());     //Apply normalization to the training data

        // ----- Load the training data -----
        SequenceRecordReader trainFeatures = new CSVSequenceRecordReader();
        SequenceRecordReader trainLabels = new CSVSequenceRecordReader();

        trainFeatures = new CSVSequenceRecordReader();
        trainFeatures.initialize(new NumberedFileInputSplit(featuresDirTrain.getAbsolutePath() + "/%d.csv", 0, lastTrainCount - lastTrainCount%miniBatchSize - 1));
        trainLabels = new CSVSequenceRecordReader();
        trainLabels.initialize(new NumberedFileInputSplit(labelsDirTrain.getAbsolutePath() + "/%d.csv", 0, lastTrainCount - lastTrainCount%miniBatchSize - 1));

        System.out.println( TAG+" "+" - trainFeatures.next().toString() - " + trainFeatures.next().toString());
        System.out.println( TAG+" "+" - trainLabels.next().toString() - " + trainLabels.next().toString());

        DataSetIterator trainData = new SequenceRecordReaderDataSetIterator(trainFeatures, trainLabels, miniBatchSize, numLabelClasses,
                true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);


        // ----- Load the test data -----
        //Same process as for the training data.
        testFeatures = new CSVSequenceRecordReader();
        testFeatures.initialize(new NumberedFileInputSplit(featuresDirTest.getAbsolutePath() + "/%d.csv", 0, lastTestCount - lastTestCount%miniBatchSize - 1));
        testLabels = new CSVSequenceRecordReader();
        testLabels.initialize(new NumberedFileInputSplit(labelsDirTest.getAbsolutePath() + "/%d.csv", 0, lastTestCount - lastTestCount%miniBatchSize - 1));

        testData = new SequenceRecordReaderDataSetIterator(testFeatures, testLabels, miniBatchSize, numLabelClasses,
                true, SequenceRecordReaderDataSetIterator.AlignmentMode.ALIGN_END);

        System.out.println( TAG+" "+" Printing traindata dataset shape");
        DataSet data = trainData.next();
        System.out.println(java.util.Arrays.toString(data.getFeatures().shape()));

        System.out.println( TAG+" "+" Printing testdata dataset shape");
        DataSet data2 = testData.next();
        System.out.println(java.util.Arrays.toString(data2.getFeatures().shape()));

        normalizer = new NormalizerStandardize();
        normalizer.fitLabel(true);

//        normalizer.fit(trainData);           //Collect the statistics (mean/stdev) from the training data. This does not modify the input data
        int index = 0;
        while(trainData.hasNext()) {
            ++index;
            System.out.println( TAG+" "+" index - " + index);
            normalizer.fit(trainData.next());     //Apply normalization to the training data
        }
159

@agibsonccc Any thoughts about the test i ran a couple of weeks ago? Should we just shelve this?

By the way, How long will you be supporting all the non-samediff code?