@adonnini could you please use setInputType instead of specifying the input sizes manually? We can automatically infer it from the beginning if you do that instead.
Also make sure to remove .nIn from each layer configuration.
Put that right before build with:
.setInputType(InputType.recurrent(..));
Ping back here if you either can’t get that working or get an error still. Usually indexing errors like that are due to a mis configured network.
Thanks!
the parameters for setInputType are:
* @param size The size of the activations
* @param timeSeriesLength Length of the input time series
Length of input time series vsaires from one feature file to the next. I
am not sure what I should put, the maximum?
The size of activations? Does this refer to the number of feature files?
I am sorry, I am asking as I figure it would take less time than trying
to figure it out by reading the coe.
Thanks,
Alex
@adonnini no not at all! The goal is to step you through 1 step at a time and make sure you understand how to configure things.
The size is the number of columns/features you have.
The timeseries length is your expected length of your time series. That can vary.
Take a look at some of the RNN reference docs for this: Recurrent Neural Network - Deeplearning4j
Also note you can also just do InputType.recurrent(numInputs) that will allow variable length time series.
I ran the code with this line added
.setInputType(InputType.recurrent(5, 645))
where 5 is the number of columns on the time series feature files, and
645 is the the length of the longest time series.
The code fails with exactly (word for word) the same failure as it did
before adding the line above (see below).
By the way, unless I completely misread the code, InputType.recurrent
requires at least two parameters. In addition, I am not sure how I would
define numInputs in an InputType.recurrent(numInputs) call. Probably, I
did not understand what you meant when you suggested I use
InputType.recurrent(numInputs).
It seems that the error is caused by the discrepancy between the index
interval (of a loop in BaseNDArray.java, I assume), and the array (of
what, time series entries?) with shape 629 which may be related to the
length of the time series (just a guess). I tried using 629 as the
length of the time series in InputType.recurrent. It made no difference.
What do you think?
Thanks,
Code execution output
------------------------------o.n.l.f.Nd4jBackend - Loaded [CpuBackend]
backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 6
o.n.l.c.n.CpuNDArrayFactory - Binary level Generic x86 optimization
level AVX512
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 6
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [30.0GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.n.l.c.n.CpuBackend - Backend build information:
GCC: “7.5.0”
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with
WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode
set to [NONE]
Exception in thread “main” java.lang.IllegalStateException: Indices are
out of range: Cannot get interval index Interval(b=0,e=640,s=1) on array
with size(0)=629. Array shape: [629], indices: [Interval(b=0,e=640,s=1)]
at org.nd4j.linalg.api.ndarray.BaseNDArray.get(BaseNDArray.java:4239)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:712)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:605)
at
org.deeplearning4j.examples.quickstart.modeling.recurrent.UCISequenceClassification.main(UCISequenceClassification.java:204)
Process finished with exit code 1
I took a quick look at the source code of MultiLayerNetwork.java, and
BaseNDArray.java. Could it be that the problem lies with this line
.layer(new
LSTM.Builder().activation(Activation.TANH).nIn(5).nOut(10).build())
in
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(123) //Random number generator seed for
improved repeatability. Optional.
.weightInit(WeightInit.XAVIER)
.updater(new Nadam())
.gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue)
//Not always required, but helps with this data set
.gradientNormalizationThreshold(0.5)
.list()
.layer(new
LSTM.Builder().activation(Activation.TANH).nIn(5).nOut(10).build())
.layer(1, new
RnnOutputLayer.Builder(LossFunctions.LossFunction.MSE)
.activation(Activation.IDENTITY).nIn(10).nOut(numLabelClasses).build())
.setInputType(InputType.recurrent(5, 645))
.build();
?
@adonnini what is your input size and shape of your input data? Something on your side there is still misconfigured. Make sure the shape of your input data matches what you’re specifying as the input.
The neural net is just configured according to that and will throw out of bounds errors if it’s not correct.
Definitely double check the shape of your data first.
OK.
If I understand the terms correctly, shape refers to the number of
features which in my case is 5. When you say size of input in your
message below, you mean the number of feature files or the size of the
feature files? I specified the latter in
setInputType(InputType.recurrent(5, 645)). If you mean the number of
feature files, I specified it here (the variable lastTrainCount):
SequenceRecordReader trainFeatures = new CSVSequenceRecordReader();
trainFeatures.initialize(new
NumberedFileInputSplit(featuresDirTrain.getAbsolutePath() + “/%d.csv”,
0, lastTrainCount));
SequenceRecordReader trainLabels = new CSVSequenceRecordReader();
trainLabels.initialize(new
NumberedFileInputSplit(labelsDirTrain.getAbsolutePath() + “/%d.csv”, 0,
lastTrainCount));
The train feature .csv files all contain 5 columns each.
Where other than the places I mention above would I specify input size
and shape of input data?
I must be missing something. Sorry for being slow on the uptake.
Hi Adam,
Some progress…
The code completed the training phase and failed during te evaluation
phase with the error log below.
The only change I made is
FROM
// .activation(Activation.IDENTITY).nIn(10).nOut(numLabelClasses).build())
TO
.activation(Activation.IDENTITY).nIn(10).nOut(1).build())
where
numLabelClasses = -1
I will take a closer look. I am not sure what is going on. What do you
think?
I don’t quite understand why there was only one iteration. I need to
find whether I can change that setting. Should I try?
Thanks,
Alex
Execution log
Sorry not seeing the log. Can you clarify “iteration”? Not sure what you mean in this context.
Below is the log
In the log there is
o.d.e.q.m.r.UCISequenceClassification - Starting training…
o.d.o.l.ScoreIterationListener - Score at iteration 0 is 82.57111206054688
Should I change the number of iterations run? Where is that setting?
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 6
o.n.l.c.n.CpuNDArrayFactory - Binary level Generic x86 optimization
level AVX512
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 6
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [30.0GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.n.l.c.n.CpuBackend - Backend build information:
GCC: “7.5.0”
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with
WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode
set to [NONE]
o.d.e.q.m.r.UCISequenceClassification - Starting training…
o.d.o.l.ScoreIterationListener - Score at iteration 0 is 82.57111206054688
o.d.o.l.EvaluativeListener - Starting evaluation nr. 1
Exception in thread “main” java.lang.IllegalArgumentException:
occurrences cannot be negative: -379967
at
org.nd4j.shade.guava.base.Preconditions.checkArgument(Preconditions.java:192)
at
org.nd4j.shade.guava.collect.AbstractMapBasedMultiset.add(AbstractMapBasedMultiset.java:254)
at org.nd4j.shade.guava.collect.HashMultiset.add(HashMultiset.java:33)
at
org.nd4j.evaluation.classification.ConfusionMatrix.add(ConfusionMatrix.java:70)
at
org.nd4j.evaluation.classification.Evaluation.eval(Evaluation.java:424)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluationHelper(MultiLayerNetwork.java:3505)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluation(MultiLayerNetwork.java:3452)
at
org.deeplearning4j.optimize.listeners.EvaluativeListener.invokeListener(EvaluativeListener.java:233)
at
org.deeplearning4j.optimize.listeners.EvaluativeListener.onEpochEnd(EvaluativeListener.java:210)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(MultiLayerNetwork.java:1779)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1688)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1675)
at
org.deeplearning4j.examples.quickstart.modeling.recurrent.UCISequenceClassification.main(UCISequenceClassification.java:213)
Process finished with exit code 1
@adonnini see what removing the listener will do first.
More progress.
Removing the listener caused the training phase to take quite a bit
longer (1-2 minutes). The code failed during the evaluation phase. I ran
into two problems.
To resolve the first problem, I followed the suggestion in the error
message (Exception in thread “main” java.lang.IllegalStateException:
Classifier evaluation using Evaluation class cannot be applied to output
layers with activation functions that are not probabilities (in range 0
to 1). Output layer type: RnnOutputLayer has activation function
ActivationIdentity. This check can be disabled using
MultiLayerNetwork.getLayerWiseConfigurations().setValidateOutputLayerConfig(false)
or ComputationGraph.getConfiguration().setValidateOutputLayerConfig(false)).
Adding to the code
MultiLayerNetwork.getLayerWiseConfigurations().setValidateOutputLayerConfig(false)
resolved the problem.
When I ran the code again. It failed with the error below.
I tried to follow the path of the code through MultiLayerNetwork without
much luck, especially as the MultiLayerNetwork line numbers in the error
log don’t seem to match well with the source on GitHub.
Does this error have to do with the content of the feature files used
for the evaluation phase? Is the error related to the line I added to
resolve the first problem (I don’t think so but…)
Thanks,
Alex
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 6
o.n.l.c.n.CpuNDArrayFactory - Binary level Generic x86 optimization
level AVX512
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 6
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [30.0GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.n.l.c.n.CpuBackend - Backend build information:
GCC: “7.5.0”
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with
WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode
set to [NONE]
o.d.e.q.m.r.UCISequenceClassification - Starting training…
o.d.e.q.m.r.UCISequenceClassification - Evaluating…
Exception in thread “main” java.lang.IllegalArgumentException:
occurrences cannot be negative: -379967
at
org.nd4j.shade.guava.base.Preconditions.checkArgument(Preconditions.java:192)
at
org.nd4j.shade.guava.collect.AbstractMapBasedMultiset.add(AbstractMapBasedMultiset.java:254)
at org.nd4j.shade.guava.collect.HashMultiset.add(HashMultiset.java:33)
at
org.nd4j.evaluation.classification.ConfusionMatrix.add(ConfusionMatrix.java:73)
at
org.nd4j.evaluation.classification.Evaluation.eval(Evaluation.java:423)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluationHelper(MultiLayerNetwork.java:3505)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluation(MultiLayerNetwork.java:3452)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3647)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3557)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3367)
at
org.deeplearning4j.examples.quickstart.modeling.recurrent.UCISequenceClassification.main(UCISequenceClassification.java:219)
Process finished with exit code 1
I checked and played round with all the input parameters I have control
over, miniBatchSize, numLabelClasses, the size parameter in
InputType.recurrent, and the size and structure of label and feature
files in both the train and test folders. It has made no difference to
the outcome. Execution still fails with the error reported below.
I am not sure what I should do next.
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 6
o.n.l.c.n.CpuNDArrayFactory - Binary level Generic x86 optimization
level AVX512
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 6
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [30.0GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.n.l.c.n.CpuBackend - Backend build information:
GCC: “7.5.0”
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
o.d.n.m.MultiLayerNetwork - Starting MultiLayerNetwork with
WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode
set to [NONE]
o.d.e.q.m.r.UCISequenceClassification - Starting training…
o.d.e.q.m.r.UCISequenceClassification - Evaluating…
Exception in thread “main” java.lang.IllegalArgumentException:
occurrences cannot be negative: -379967
at
org.nd4j.shade.guava.base.Preconditions.checkArgument(Preconditions.java:192)
at
org.nd4j.shade.guava.collect.AbstractMapBasedMultiset.add(AbstractMapBasedMultiset.java:254)
at org.nd4j.shade.guava.collect.HashMultiset.add(HashMultiset.java:33)
at
org.nd4j.evaluation.classification.ConfusionMatrix.add(ConfusionMatrix.java:73)
at
org.nd4j.evaluation.classification.Evaluation.eval(Evaluation.java:423)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluationHelper(MultiLayerNetwork.java:3505)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluation(MultiLayerNetwork.java:3452)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3647)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3557)
at
org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:3367)
at
org.deeplearning4j.examples.quickstart.modeling.recurrent.UCISequenceClassification.main(UCISequenceClassification.java:223)
Process finished with exit code 1
@adonnini this all feels horribly unnecessary and shouldn’t be remotely this complicated. Could you please setup a github reproducer I can download and run? All of these workarounds and random problems are far from the norm and should not be happening. I have a feeling there’s just something you’re not telling me with the data. It’s probably easier for me to just fix this instead.
Well, not surprisingly, there is a simple explanation/resolution for the
problems. It’s not that I did not tell you everything about the data.
I made a (big) mistake showing that I still have a lot to learn about
how dl4j works.
When I modified UCISequenceClassification.java following the example in
SingleTimestepRegressionExample.java, which you pointed me to, initially
the only change I made was to the configuration of the network failing
to also change the definition of the normalizer (and related code),
etc., etc.
To make a long story short, after I made all the necessary changes,
execution of UCISequenceClassification to build, train and test a time
series based regression completes execution successfully.
The next challenge is to understand whether the output makes any sense.
Currently UCISequenceClassification is configured to
System.out.println(evaluation.stats());
The code is also configured to plot train data, actual test data, and
predicted test data. My reading of the plots makes me think the network
as defined is not doing what I would like t to do, not surprisingly.
What do you think should be my next step?
Thanks
One other point. this comment in SingleTimestepRegressionExample caught
my eye
“//Run regression evaluation on our single column input” – not sure
exactly which input the comment refers to
In my case fature files have 5 columns and label files have one column.
Should my next step to modify the code to take 5-column input?
It’s not as simple as changing
RegressionEvaluation evaluation = new RegressionEvaluation(1);
to
RegressionEvaluation evaluation = new RegressionEvaluation(5);
I tried. The failure is
Exception in thread “main” java.lang.IllegalArgumentException: Number of
the columns of labels and predictions must match specification (5). Got
1 and 1
at
org.nd4j.evaluation.regression.RegressionEvaluation.eval(RegressionEvaluation.java:246)
at
org.nd4j.evaluation.regression.RegressionEvaluation.eval(RegressionEvaluation.java:216)
at
org.nd4j.evaluation.BaseEvaluation.evalTimeSeries(BaseEvaluation.java:285)
at
org.nd4j.evaluation.BaseEvaluation.evalTimeSeries(BaseEvaluation.java:272)
at
org.deeplearning4j.examples.quickstart.modeling.recurrent.UCISequenceClassification.main(UCISequenceClassification.java:260)
I have been trying to make some sense of the output with (very) limited
success. I don’t think it is meaningful.
I think a (main?) reason for the problem with the output is that the
example I am basing mmy code on was based on a time series which is
pretty different from mine
(https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/).
I am not having much success in trying to modify the code to adjust for
my feature and label input files.
I would appreciate any suggestions.
Thanks
@adonnini yeah one thing I mentioned was potentially just building a 2d classifier first using DenseLayers. Glad you navigated it! I wanted a reproducer so I could help you a bit faster.
Keep in mind I already mentioned this: Your problem you want to solve doesn’t sound like a classification problem but a regression one. A lot of time seems like it’s been wasted because you jumped straight in and didn’t try to understand the very fundamental things like what an ndarray is, what classification is, what regression is.
It was just a bit hard with the go around. I couldn’t really get a complete picture of your issue.
If you don’t have time series don’t use LSTMs. It won’t really help you. You could build a simple 2d classifier as a baseline. Stepping back a bit it might be the way you’re modeling it.
Could you still put up a reproducer? Feel free to DM me if you don’t want it public.
Otherwise feel free to describe how you’re modeling it and I can try to point something out. A sample line from one of your files and what you are doing will be needed though.
Thanks Adam,
I do have a time series! Please see the attached sample feature file
The columns in all feature files are “id,latitude,longitude,track_id,time”
Feature files of varying size all with the same structure. Here is a
sample entry from a feature file:
-10.8694503931813 -37.0952764284214 38090 2016-01-03\00:58:23
Labels are in separate files. There is an equal number of feature and
label files.
Label file entries have a single column (track_id). Here is a sample
entry from a label file:
38082
Do you need any other information?
One thing to note. Although I have separate feature and label files (as
defined in UCISequenceClassification), I still kept the track_id values
in the feature files. Could that be causing a problem? It’s easy to
remove the track_id column from the feature files.
When you say “free to describe how you’re modeling it”, I am not sure
what you mean, and perhaps herein lies the problem. Other than
defining/constructing feature files and label files, I do not do
anything to “model” my data. I use the “modeling” as defined in the code
I am using. Can you give me an explicit example of what you mean?
I will try to assemble a reproducer. Beware, my code is not pretty, to
say the least.
Thanks