Ok. Thanks. Is this a change? The addition of the boolean ```inverted```?
In any event, what does ```inverted``do? Is the default value```true```or```false```
Thanks
Inverted means the dropout probability you want to use is 1 - instead of the probability you specify. If it’s 0.5 it would be the same. If you specify inverted true with 0.9 it would be 1 - 0.9.
Inverted dropout is common in some use cases. You can probably just leave it as false.
Thanks. I took a look at the source code for dropout and came to the same conclusion.
Now, execution fails where it didn’t previously. I don’t think this is related to any of the changes in the new release but I wanted to run it by you, just in case.
Below, you will find the section of the code where execution fails with the following error:
Exception in thread "main" java.lang.RuntimeException: No data was added, statistics cannot be determined
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:161)
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:85)
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:112)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.prepareTrainingAndTestData(LocationNextNeuralNetworkV7_04.java:2163)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.sameDiff3(LocationNextNeuralNetworkV7_04.java:228)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.main(LocationNextNeuralNetworkV7_04.java:221)
the line where the failure occurs in the code below is:
normalizer.fit(trainData); //Collect the statistics (mean/stdev) from the training data. This does not modify the input data
What really puzzles me is that the output of the print statements in the code below is as expected (and as it was previously).
Please let me know if, as I doubt, you think this may be related to recent changes and if you have any other thoughts.
Thanks,
Alex
CODE
trainData.reset();
testData.reset();
System.out.println( TAG+" "+" prepareTrainingAndTestData - Printing traindata dataset shape - 1 - ");
DataSet data = trainData.next();
System.out.println(Arrays.toString(data.getFeatures().shape()));
System.out.println( TAG+" "+" prepareTrainingAndTestData - Printing testdata dataset shape - 1 - ");
DataSet data2 = testData.next();
System.out.println(Arrays.toString(data2.getFeatures().shape()));
normalizer = new NormalizerStandardize();
normalizer.fitLabel(true);
normalizer.fit(trainData); //Collect the statistics (mean/stdev) from the training data. This does not modify the input data
trainData.reset();
testData.reset();
Quick update.
I added this statement
System.out.println( TAG+" "+" prepareTrainingAndTestData - Printing trainData.next().get(0).getFeatures() - "+trainData.next().get(0).getFeatures());
```
The result was
```
LocationNextNeuralNetworkV7_04.java prepareTrainingAndTestData - Printing trainData.next().get(0).getFeatures() - [[[ 42.3547, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ -71.1776, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 1.7533e12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 1.0000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]]
```
To me this says that trainData does have data making the error statement
more puzzling.
@adonnini sorry again please ping me. I only react to forum notifications. Try ignoring the eval?
Can you give any updates to what you tried? Otherwise I can dig in to this more this week.
No Problem . I know you are busy and have higher priorities to take care of.
Below, you will find the section of the code where execution fails with the following error:
Exception in thread "main" java.lang.RuntimeException: No data was added, statistics cannot be determined
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:161)
at org.nd4j.linalg.dataset.api.preprocessor.stats.DistributionStats$Builder.build(DistributionStats.java:85)
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:112)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.prepareTrainingAndTestData(LocationNextNeuralNetworkV7_04.java:2163)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.sameDiff3(LocationNextNeuralNetworkV7_04.java:228)
at trajectorypredictiontransformer.LocationNextNeuralNetworkV7_04.main(LocationNextNeuralNetworkV7_04.java:221)
the line where the failure occurs in the code below is:
normalizer.fit(trainData); //Collect the statistics (mean/stdev) from the training data. This does not modify the input data
What really puzzles me is that the output of the print statements in the code below is as expected (and as it was previously).
Please let me know if, as I doubt, you think this may be related to recent changes and if you have any other thoughts.
Thanks,
Alex
CODE
trainData.reset();
testData.reset();
System.out.println( TAG+" "+" prepareTrainingAndTestData - Printing traindata dataset shape - 1 - ");
DataSet data = trainData.next();
System.out.println(Arrays.toString(data.getFeatures().shape()));
System.out.println( TAG+" "+" prepareTrainingAndTestData - Printing testdata dataset shape - 1 - ");
DataSet data2 = testData.next();
System.out.println(Arrays.toString(data2.getFeatures().shape()));
normalizer = new NormalizerStandardize();
normalizer.fitLabel(true);
normalizer.fit(trainData); //Collect the statistics (mean/stdev) from the training data. This does not modify the input data
trainData.reset();
testData.reset();
Below is an update with additional information
@agibsonccc when you say “try ignoring the “eval”, do you mean commenting this line:
RegressionEvaluation eval = new RegressionEvaluation(2);
```
The ```eval``` variable is never used
Thanks,
Alex
@adonnini I’m going off only very vague recollection here but you should be passing that to a listener which is calling that. I mainly want to reduce complexity first then we can narrow this down.
@agibsonccc I am sorry. I don’t understand your suggestion. Could you please try to be a little more specific to at least give me a starting point?
The code where the failure occurs is very old code which I used on all models Not just this one. It has never failed.
In all the models I have build, which have worked (before and after sameDiff) I never used a listener except for the UI.
The failure occurs while setting up test and training data. Nothing to do with the UI, even remotely.
What am I missing from your message?
Thanks
@adonnini your models uses listeners right. You’re creating a regression evaluation object. It has to be used somewhere by something.
I remember there being a listener that invokes the eval method on each call of fit(…) when the model updates its weights to show the current statistics like the f1 score or the RMSE or something similar.
When that happens, a calculation is run for determining the metric used by the evaluation object. That is what’s throwing an error from what I’m seeing. My first thinking was to just let JUST the model run and comment out some steps.
That would mean comment out where you’re passing the regression eval object to the listener OR commenting out eval.eval.
eval.eval is already commented everywhere in my code.
The call to fit where execution fails takes place way before I define the training configuration (see below).
I am sorry for asking. Is it a coincidence that execution fails where it NEVER failed before with any model after you released the new snapshot?
training configuration
TrainingConfig config = new TrainingConfig.Builder()
// .l2(1e-4) //L2 regularization
// .l2(0.001)
// .updater(new Adam(learningRate)) //Adam optimizer with specified learning rate
.updater(new Nesterovs(0.01, 0.9))
// .updater(new Nesterovs(0.0001, 0.9))
.dataSetFeatureMapping("input") //DataSet features array should be associated with variable "input"
.dataSetLabelMapping("label") //DataSet label array should be associated with variable "label"
// .minimize("lossLOG")
// .minimize(lossMSE.name())
// .minimize("lossMSE")
// .trainEvaluation(outReduced.name(), 0, evaluation) // add a training evaluation
.trainEvaluation(outReduced.name(), 0, evaluation) // add a training evaluation
// .trainEvaluation("outReduced"+" - "+mRandomNumericalId, 0, evaluation) // add a training evaluation
// .trainEvaluation("out", 0, evaluation) // add a training evaluation
.build();
@adonnini eval.eval is called internally. Sorry again please ping me. I rely on email to check the forums as a reminder. I also just got back from a break with family.
@agibsonccc Welcome back. I am not sure what you are asking me to do. Were you able to read the last two messages I sent above?
I mentioned that eval.eval is already commneted out in my code because you asked me to comment it out in your last message above. Did I misunderstand your suggestion?
@adonnini apologies. I mainly mean just make sure to at me so I see this.
I was replying to your comment about eval not being used. It’s being passed in and should normally be used by a listener. Make sure you comment out all usage including parameter passing.
My only goal is to eliminate anything that’s not needed first to ensure that we’re just debugging a running model and not extra processes.
Please bear with me.
I commented out any use of listeners.
I have left enabled this:
RegressionEvaluation eval = new RegressionEvaluation(2);
and this:
evaluation = new Evaluation();
double learningRate = 1e-3;
TrainingConfig config = new TrainingConfig.Builder()
.updater(new Nesterovs(0.01, 0.9))
.dataSetFeatureMapping("input") //DataSet features array should be associated with variable "input"
.dataSetLabelMapping("label") //DataSet label array should be associated with variable "label"
.trainEvaluation(outReduced.name(), 0, evaluation) // add a training evaluation
.build();
Other than the two places above nothing with eval or evaluation is enabled.
I ran the code. It failed in exactly the same way. the failure occurs well before excution reaches either of the two places above.
Please let me know what I should do next.
Thanks
@agibsonccc I know you are busy with paying customers, sorry to bug you. I would really like to move on and try to resolve this issue. Did you get a chance to take a look at my latest message above?
Thanks
@adonnini thanks for pinging me. You need to get rid of the evaluation usage everywhere.
Sorry when I give you hints I’m expecting you to generalize a bit or ask me clarifications.
In this case when I said “comment out the listener usage” you still left in one example eval with the training evaluation.
The point as I specified to you earlier was to remove sources of errors till we can narrow down a core issue.
Anyways..looking at this…it seems like the normalizer is running out of data? Your running mean was null in the distribution stats, see here:
As for why you’d have to go in to the debugger and trace that. See what lead to that.Your issue here should be fairly straightforward.
Debugging wise, let me try to teach you how to fend for yourself a bit here. I would REALLY encourage you to use the tooling available in your IDE. Both eclipse and intellij both have free versions with a debugger that automatically download source code (which I publish for releases) that will show you exactly where something is going wrong.
Both IDEs also allow you to search for given classes and place breakpoints. Debugger screenshots alongside different stack frames at different points of execution will help me A LOT more in helping you. There’s only so much I can guess from the small bits of stack traces you give me.
Thanks @agibsonccc I will follow your instructions and not bother you (at least for a while).
I am sorry but you still not have addressed the fact that before you made the recent changes/update related to flatbuffers to your software, my code worked without problems where it is failing now, and it did so for years.
Thanks
@adonnini sorry about the flatbuffers changes. I guarantee you there’s something else going on.
Im only trying to help you narrow it down. I just don’t have a lot to go on. That’s why I was trying to give you more tools to work with . I don’t think asking for some basic reproduction steps is going too far.
II had to create my own reproducer in order to even make your test case work. I did work with you there. Try to meet me somewhere here as well.
Luckily, hey I had a model that saved > 2g! That by itself was a lot of work and will be a great asset that the project needed anyways. I’d appreciate if you didn’t ignore the effort that went in to that, let alone the fact that I found a way to work around not having the initial test case/reproducer that I thought would have made my life a bit easier to start.
It’s not about “bothering me” I’m just not going to try to fully dive in to your code base. Thjere’s a lot of commented code, the flow isn’t very intuitive for me. I asked you for stripped down reproducers and other things a few times now I believe and you weren’t really able to.
I’m only doing what any open source maintainer offering free help to maintain their sanity does: helping people help themselves and setting a clear standard to make the support burden at scale easier. I’d appreciate it if you’d consider trying to take some of the steps I’m mentioning rather than brushing it off as not bothering me. Thanks!
Literally all java developers do these basic things every day like using an ide, debugging code, clicking “download source”. These are not exactly new things. I don’t think asking you to use industry standard tools is asking for much.