Exception in thread "ADSI prefetch thread" Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid indices: cannot get [1,5] from a [20, 5] NDArray

DL4J
1

I have dataset with 7 columns of features and 1 label. Label is multy class: 1, 2 ,3 , 4 or 5 value.

I uloaded dataset and set NN paramaters next:

int seed = 123;
        double learningRate = 0.01;
        int batchSize = 20;
        int nEpochs = 7;

        int numInputs = 7;
        int numOutputs = 5;
        int numHiddenNodes = 20;

        //dataLocalPath = ''
        //Load the training data:
        RecordReader rr = new CSVRecordReader();
        rr.initialize(new FileSplit(new File( "scs_TRAIN.csv")));
        DataSetIterator trainIter = new RecordReaderDataSetIterator(rr, batchSize, 7,5);

        //Load the test/evaluation data:
        RecordReader rrTest = new CSVRecordReader();
        rrTest.initialize(new FileSplit(new File("scs_TEST.csv")));
        DataSetIterator testIter = new RecordReaderDataSetIterator(rrTest, batchSize, 7,5);

        MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
            .seed(seed)
            .weightInit(WeightInit.XAVIER)
            .updater(new Nesterovs(learningRate, 0.9))
            .list()
            .layer(new DenseLayer.Builder().nIn(numInputs).nOut(numHiddenNodes)
                .activation(Activation.RELU)
                .build())
            .layer(new OutputLayer.Builder(LossFunction.NEGATIVELOGLIKELIHOOD)
                .activation(Activation.SOFTMAX)
                .nIn(numHiddenNodes).nOut(numOutputs).build())
            .build();

And after:

o.d.o.l.ScoreIterationListener - Score at iteration 9200 is 0.00621582604944706
o.d.o.l.ScoreIterationListener - Score at iteration 9300 is 0.001661047525703907

iterations I have this Error:

Exception in thread "ADSI prefetch thread" Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid indices: cannot get [1,5] from a [20, 5] NDArray
	at org.nd4j.linalg.dataset.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:437)
Caused by: java.lang.IllegalArgumentException: Invalid indices: cannot get [1,5] from a [20, 5] NDArray
	at org.nd4j.linalg.api.shape.Shape.getOffsetUnsafe(Shape.java:1093)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.putScalar(BaseNDArray.java:1421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesBatched(RecordReaderMultiDataSetIterator.java:421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:360)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:333)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:213)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:378)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:453)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:85)
	at org.nd4j.linalg.dataset.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:411)
java.lang.IllegalArgumentException: Invalid indices: cannot get [1,5] from a [20, 5] NDArray
	at org.nd4j.linalg.api.shape.Shape.getOffsetUnsafe(Shape.java:1093)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.putScalar(BaseNDArray.java:1421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesBatched(RecordReaderMultiDataSetIterator.java:421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:360)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:333)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:213)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:378)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:453)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:85)
	at org.nd4j.linalg.dataset.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:411)

Process finished with exit code 1

What could be the reason of it ?

Format of my dataset is:

time,gFx,gFy,gFz,wx,wy,wz,speed,label
0.0,-0.2727,0.0442,0.9382,-0.0053,-0.0077,-0.0331,0.0,1.0
0.0100000070759929,-0.2727,0.0442,0.9382,-0.0053,-0.0077,-0.0331,0.0,1.0
2

Could you try using: https://deeplearning4j.org/api/latest/org/deeplearning4j/datasets/iterator/AsyncShieldDataSetIterator.html

There might be a bug we’re not seeing with your environment.

3

I made next:>

RecordReader rr = new CSVRecordReader();
rr.initialize(new FileSplit(new File( “scs_TRAIN.csv”)));
DataSetIterator trainIter = new RecordReaderDataSetIterator(rr, batchSize, 7,5);
DataSetIterator trainI = new AsyncShieldDataSetIterator(trainIter);
//Load the test/evaluation data:
RecordReader rrTest = new CSVRecordReader();
rrTest.initialize(new FileSplit(new File(“scs_TEST.csv”)));
DataSetIterator testIter = new RecordReaderDataSetIterator(rrTest, batchSize, 7,5);
DataSetIterator testI = new AsyncShieldDataSetIterator(testIter);

And also have an error:

o.d.o.l.ScoreIterationListener - Score at iteration 9100 is 3.191389376297593E-4
o.d.o.l.ScoreIterationListener - Score at iteration 9200 is 0.0023097645491361616
o.d.o.l.ScoreIterationListener - Score at iteration 9300 is 9.434095583856106E-4
Exception in thread "main" java.lang.IllegalArgumentException: Invalid indices: cannot get [1,5] from a [20, 5] NDArray
	at org.nd4j.linalg.api.shape.Shape.getOffsetUnsafe(Shape.java:1093)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.putScalar(BaseNDArray.java:1421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesBatched(RecordReaderMultiDataSetIterator.java:421)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:360)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:333)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:213)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:378)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:453)
	at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:85)
	at org.deeplearning4j.datasets.iterator.AsyncShieldDataSetIterator.next(AsyncShieldDataSetIterator.java:166)
	at org.deeplearning4j.datasets.iterator.AsyncShieldDataSetIterator.next(AsyncShieldDataSetIterator.java:35)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(MultiLayerNetwork.java:1684)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1636)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1623)
	at org.deeplearning4j.examples.quickstart.modeling.feedforward.classification.scs.main(scs.java:104)

Process finished with exit code 1
4

Thanks a lot for confirming then. Could you give me something end to end reproducible to look at?

5

Do you need all code example ?
Wait a second.

6

So, there is my java class: package org.deeplearning4j.examples.quickstart.modeling.feedforward.classificati - Pastebin.com

8

Hope I found mistake.
Will reply what the reason later.

9

So, I just not sure that all my data is correctly.
It has some broken data that is meaningless in training stage.