Training CNN error / CNN text classification

DL4J
21

There is no dataformat method for 1d layer

Edit

I chose 87 instead of 1 as nIn.

Exception in thread "main" java.lang.IllegalArgumentException: Labels and preOutput must have equal shapes: got shapes [150, 2, 255] vs [150, 2]
at org.nd4j.common.base.Preconditions.throwEx(Preconditions.java:636)
at org.nd4j.linalg.lossfunctions.impl.LossMCXENT.computeGradient(LossMCXENT.java:149)
at org.deeplearning4j.nn.layers.BaseOutputLayer.getGradientsAndDelta(BaseOutputLayer.java:175)
at org.deeplearning4j.nn.layers.BaseOutputLayer.backpropGradient(BaseOutputLayer.java:147)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.calcBackpropGradients(MultiLayerNetwork.java:1946)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2761)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2704)
22

Oh the many special cases :sweat_smile: Try .rnnDataFormat(NWC) then.

That makes sense, as you’ve got a sequence of labels the way it is currently set up.

You can set LabelLastTimeStepPreProcessor on your dataset iterator so it has the right form.

23

It runs. Nice!

========================Evaluation Metrics========================
 # of classes:    2
 Accuracy:        1,0000
 Precision:       0,0000
 Recall:          0,0000
 F1 Score:        0,0000
Precision, recall & F1: reported for positive class (class 1 - "1") only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [1]
Warning: 1 class was never predicted by the model and was excluded from average recall
Classes excluded from average recall: [1]

=========================Confusion Matrix=========================
    0    1
-----------
 1535    0 | 0 = 0
    0    0 | 1 = 1

Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times
==================================================================

But there is an issue, with the recognition of the classes/labels. It can have several reasons. What I have observed is that in my final schema there are two columns for the labels. And I specify as label index 0 in SequenceRecordReaderDataSetIterator. Is there anything else I need to adjust in my schema?

   "columns" : [ {
    "@class" : "org.datavec.api.transform.metadata.IntegerMetaData",
    "name" : "label[good]",
    "minAllowedValue" : 0,
    "maxAllowedValue" : 1
  }, {
    "@class" : "org.datavec.api.transform.metadata.IntegerMetaData",
    "name" : "label[block]",
    "minAllowedValue" : 0,
    "maxAllowedValue" : 1
  }, {
    "@class" : "org.datavec.api.transform.metadata.IntegerMetaData",
    "name" : "char[0]",
    "minAllowedValue" : 0,
    "maxAllowedValue" : 1
  }, ....

Also, I am currently padding with zeros only. I don’t know what is the best solution.

I have updated the gist again.

24

The trim or pad transform may be a bad choice in your case, as you are pulling your labels from the last step of the sequence and that transform adds to the end of the sequence.

For now, I’d suggest you pad it yourself when creating the data.

If you have a high number of short sequences, you probably have introduced a bias towards label 0 with the way it was padded, and that may explain why you are seeing that result.

25

Thank you very much, I think it works quite well now.

# of classes:    2
Accuracy:        0,9707
Precision:       0,9283
Recall:          0,9756
F1 Score:        0,9514


  0    1
-----------
1052   32 | 0 = 0
  28  423 | 1 = 1

My only question now is how to make a prediction about my model. I have to pass an INDArray for this. How do I have to build the array? Can I also create it via my schema and how do I handle the label column?

I had still changed with my schema that I transform the label as Categorical to Integer, since it has then only one column. Can I also make it that I leave the String/Categorical in it? That would be a bit nicer in the evaluation, but not very important. Because it didn’t work for me, there was always a NumberFormatException.

Exception in thread "ADSI prefetch thread" java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "block"
at org.nd4j.linalg.dataset.AsyncDataSetIterator$AsyncPrefetchThread.run(AsyncDataSetIterator.java:437)
Caused by: java.lang.NumberFormatException: For input string: "block"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.datavec.api.writable.Text.toInt(Text.java:610)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesSequence(RecordReaderMultiDataSetIterator.java:690)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:369)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:333)
26

The INDArray for it will have the same general shape: [batchSize, alphabetSize, seqLength]. If you have only a single example you want to predict for, the shape will then be [1, 87, 255] in your case.

It is up to you how you create it.

You can create it directly with Nd4j.zeros(1, 87, 255) and then set the ones for your characters appropriately.

Or you can point your DataSetIterator builder to a folder where you have your example in a file, and then apply the same kind of transformation as you do for your training data. You can then use the dataset iterator to get the dataset (iter.next()), and then get the features from the dataset (ds.getFeatures()). In this case you’d need to extend your builder a bit, so it sets a fake label, the actual value doesn’t matter to much, because we aren’t going to use it.

How do you expect it to do calculations based on strings? :wink:

The model works with numbers only, so your label needs to be transformed into a number. With only two labels, you can map them to 0 and 1, but once you’ve got more than two, you have to use a one-hot encoding for your labels.

For this reason you get as many columns as you have labels, and with a Softmax activation function and multi class cross entropy as the loss function it calculates a probability distribution for your labels when you try to predict an outcome.

Just pass a list of your labels to the evaluate function. Just make sure they are in the correct order, as it is mapped entirely by position.

model.evaluate(testIter, Arrays.asList("good", "block"));
27

Thank you very much, that is exactly what I meant.

I have now run it with Cuda and have noticed that it runs worse with Cuda than with my CPU. I have already reduced the batch size to adjust the performance, but when I now try to reduce the maximum length I get an OutOfMemory Error. The funny thing is that it works with 255 but not with e.g. 200 as length.

https://gist.github.com/ForceUpdate1/5d4c71517873ef421dde1d6b22ff7cdc

My GPU: Geforce GTX 1070
My CPU: Intel I7-9700K

Exception in thread "main" java.lang.RuntimeException: Error during neural network forward pass
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.calcBackpropGradients(MultiLayerNetwork.java:2030)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2761)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2704)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:170)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(MultiLayerNetwork.java:1715)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1636)
Caused by: java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (170M + 44388M) > maxBytes (7262M)
	at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:678)
	at org.deeplearning4j.cuda.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:125)
	at org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.backpropGradient(CudnnConvolutionHelper.java:325)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.backpropGradient(ConvolutionLayer.java:186)
	at org.deeplearning4j.nn.layers.convolution.Convolution1DLayer.backpropGradient(Convolution1DLayer.java:146)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.calcBackpropGradients(MultiLayerNetwork.java:1946)
	... 9 more

[main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
[main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 32
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Windows 10]
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [8]; Memory: [7,1GB];
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [CUBLAS]
[main] INFO org.nd4j.linalg.jcublas.JCublasBackend - ND4J CUDA build version: 10.2.89
[main] INFO org.nd4j.linalg.jcublas.JCublasBackend - CUDA device 0: [NVIDIA GeForce GTX 1070]; cc: [6.1]; Total memory: [8589934592]
[main] INFO org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
Layer helper: org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper
28

That certainly looks weird, for some reason it tries to allocate over 44GB. Are you sure that you didn’t set it to something weirdly large by mistake?

Also, what does the performance listener tell you when it does run?

29

I don’t change anything except that I add the maven dependency for Cuda.

CPU:

[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 15; iteration time: 417 ms; samples/sec: 306,954; batches/sec: 2,398; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 16 is 1.056962203589254
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 16; iteration time: 431 ms; samples/sec: 296,984; batches/sec: 2,320; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 17 is 1.0990315867560574
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 17; iteration time: 412 ms; samples/sec: 310,680; batches/sec: 2,427; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 18 is 1.067337314499658
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 18; iteration time: 443 ms; samples/sec: 288,939; batches/sec: 2,257; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 19 is 1.0821172207474876
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 19; iteration time: 424 ms; samples/sec: 301,887; batches/sec: 2,358; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 20 is 1.059284853091354
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 0 ms; iteration 20; iteration time: 419 ms; samples/sec: 305,489; batches/sec: 2,387;

GPU:

[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 301 ms; iteration 10; iteration time: 433 ms; samples/sec: 295,612; batches/sec: 2,309; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 11 is 1.1342023435993043
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 219 ms; iteration 11; iteration time: 418 ms; samples/sec: 306,220; batches/sec: 2,392; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 12 is 1.1255240634763897
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 156 ms; iteration 12; iteration time: 204 ms; samples/sec: 627,451; batches/sec: 4,902; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 13 is 1.1135302663130682
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 330 ms; iteration 13; iteration time: 380 ms; samples/sec: 336,842; batches/sec: 2,632; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 14 is 1.0928231754740074
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 301 ms; iteration 14; iteration time: 349 ms; samples/sec: 366,762; batches/sec: 2,865; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 15 is 1.1150627790913594
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 300 ms; iteration 15; iteration time: 348 ms; samples/sec: 367,816; batches/sec: 2,874; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 16 is 1.1025744730292892
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 305 ms; iteration 16; iteration time: 353 ms; samples/sec: 362,606; batches/sec: 2,833; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 17 is 1.10063111863199
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 331 ms; iteration 17; iteration time: 385 ms; samples/sec: 332,468; batches/sec: 2,597; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 18 is 1.0874097650678176
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 314 ms; iteration 18; iteration time: 361 ms; samples/sec: 354,571; batches/sec: 2,770; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 19 is 1.069476212327744
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 313 ms; iteration 19; iteration time: 362 ms; samples/sec: 353,591; batches/sec: 2,762; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 20 is 1.0825679057827262
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 299 ms; iteration 20; iteration time: 439 ms; samples/sec: 291,572; batches/sec: 2,278; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 21 is 1.0689636559527844
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 215 ms; iteration 21; iteration time: 434 ms; samples/sec: 294,931; batches/sec: 2,304; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 22 is 1.0424455383996136
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 151 ms; iteration 22; iteration time: 195 ms; samples/sec: 656,410; batches/sec: 5,128;

GPU maxLength=128

[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 126 is 0.5322405647156705
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 146 ms; iteration 126; iteration time: 175 ms; samples/sec: 731,429; batches/sec: 5,714; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 127 is 0.5878828953263133
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 149 ms; iteration 127; iteration time: 177 ms; samples/sec: 723,164; batches/sec: 5,650; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 128 is 0.5516416211974764
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 164 ms; iteration 128; iteration time: 192 ms; samples/sec: 666,667; batches/sec: 5,208; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 129 is 0.5697628848869166
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 152 ms; iteration 129; iteration time: 179 ms; samples/sec: 715,084; batches/sec: 5,587; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 130 is 0.5671227672971232
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 150 ms; iteration 130; iteration time: 245 ms; samples/sec: 522,449; batches/sec: 4,082; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 131 is 0.5923680057108589
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 85 ms; iteration 131; iteration time: 225 ms; samples/sec: 568,889; batches/sec: 4,444; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 132 is 0.4552335921238525
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 39 ms; iteration 132; iteration time: 73 ms; samples/sec: 1753,425; batches/sec: 13,699; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 133 is 0.5849684142028828
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 142 ms; iteration 133; iteration time: 171 ms; samples/sec: 748,538; batches/sec: 5,848; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 134 is 0.5591012373442021
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 147 ms; iteration 134; iteration time: 175 ms; samples/sec: 731,429; batches/sec: 5,714; 
[main] ERROR org.deeplearning4j.util.CrashReportingUtil - >>> Out of Memory Exception Detected. Memory crash dump written to: C:\Users\Marco\Documents\intellij\ProseminarTest\dl4j-memory-crash-dump-1620395836156_1.txt
[main] WARN org.deeplearning4j.util.CrashReportingUtil - Memory crash dump reporting can be disabled with CrashUtil.crashDumpsEnabled(false) or using system property -Dorg.deeplearning4j.crash.reporting.enabled=false
[main] WARN org.deeplearning4j.util.CrashReportingUtil - Memory crash dump reporting output location can be set with CrashUtil.crashDumpOutputDirectory(File) or using system property -Dorg.deeplearning4j.crash.reporting.directory=<path>
Exception in thread "main" java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (130M + 32640M) > maxBytes (7262M)
	at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:678)
	at org.deeplearning4j.cuda.BaseCudnnHelper$DataCache.<init>(BaseCudnnHelper.java:125)
	at org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper.preOutput(CudnnConvolutionHelper.java:516)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.preOutput(ConvolutionLayer.java:401)
	at org.deeplearning4j.nn.layers.convolution.Convolution1DLayer.preOutput(Convolution1DLayer.java:181)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.activate(ConvolutionLayer.java:489)
	at org.deeplearning4j.nn.layers.convolution.Convolution1DLayer.activate(Convolution1DLayer.java:228)
	at org.deeplearning4j.nn.layers.AbstractLayer.activate(AbstractLayer.java:258)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.ffToLayerActivationsInWs(MultiLayerNetwork.java:1134)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2746)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2704)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:170)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(MultiLayerNetwork.java:1715)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1636)

Crashreport
https://gist.github.com/ForceUpdate1/bf9a05faeede4854e27cd274aa50f8dc

30

The speed problem is then most likely coming from the ETL side of things.

As a sanity check, I suggest you use the CPU backend and save your dataset objects (iterate manually though your dataset iterator) into files and load them when using the GPU backend.

31

To fix the issue with the memory?
And what is the best way to do that?

32

Both for memory and speed. I think that most of the time is spent in loading the data when you are using the gpu, so we’ll make the data loading as fast as possible :slight_smile:

There are two ways to do it.

The manual way where you literally just iterate through your iterator and call dataset.save to write it, and then manually load it with dataSet.load.

The somewhat more automatic way is to use the CachingDataSetIterator to wrap your iterator, and tell it to use InFileDataSetCache for its cache. Then you iterate through your entire iterator once, and it should create a full cache set of your data, and it will not need to pull the data from your iterator anymore.

33

Thank you, it works like this, but there is still the same issue.
The performance of my GPU is also only used between 1-2%.

I have updated the gist

[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 496 is 0.10585051029733583
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 206 ms; iteration 496; iteration time: 238 ms; samples/sec: 537,815; batches/sec: 4,202; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 497 is 0.14658270841410836
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 204 ms; iteration 497; iteration time: 236 ms; samples/sec: 542,373; batches/sec: 4,237; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 498 is 0.1480988817494474
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 206 ms; iteration 498; iteration time: 238 ms; samples/sec: 537,815; batches/sec: 4,202; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 499 is 0.1930042161253558
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 207 ms; iteration 499; iteration time: 237 ms; samples/sec: 540,084; batches/sec: 4,219; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 500 is 0.1825534638800544
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 204 ms; iteration 500; iteration time: 307 ms; samples/sec: 416,938; batches/sec: 3,257; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 501 is 0.29321374873134465
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 203 ms; iteration 501; iteration time: 354 ms; samples/sec: 361,582; batches/sec: 2,825; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 502 is 0.1227437764217122
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 202 ms; iteration 502; iteration time: 232 ms; samples/sec: 551,724; batches/sec: 4,310; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 503 is 0.24002059704617631
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 204 ms; iteration 503; iteration time: 235 ms; samples/sec: 544,681; batches/sec: 4,255; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 504 is 0.20751082163978157
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 201 ms; iteration 504; iteration time: 230 ms; samples/sec: 556,522; batches/sec: 4,348; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 505 is 0.13938896287127706
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 199 ms; iteration 505; iteration time: 229 ms; samples/sec: 558,952; batches/sec: 4,367; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 506 is 0.22289921273349647
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 198 ms; iteration 506; iteration time: 228 ms; samples/sec: 561,404; batches/sec: 4,386; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 507 is 0.11233574892995846
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 203 ms; iteration 507; iteration time: 233 ms; samples/sec: 549,356; batches/sec: 4,292; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 508 is 0.19903439316610627
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 201 ms; iteration 508; iteration time: 231 ms; samples/sec: 554,113; batches/sec: 4,329; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 509 is 0.2034679180525244
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 201 ms; iteration 509; iteration time: 232 ms; samples/sec: 551,724; batches/sec: 4,310; 
[main] INFO org.deeplearning4j.optimize.listeners.ScoreIterationListener - Score at iteration 510 is 0.21685668483286896
[main] INFO org.deeplearning4j.optimize.listeners.PerformanceListener - ETL: 199 ms; iteration 510; iteration time: 305 ms; samples/sec: 419,672; batches/sec: 3,279; 
[main] ERROR org.deeplearning4j.util.CrashReportingUtil - >>> Out of Memory Exception Detected. Memory crash dump written to: ProseminarTest\dl4j-memory-crash-dump-1620553291249_1.txt
[main] WARN org.deeplearning4j.util.CrashReportingUtil - Memory crash dump reporting can be disabled with CrashUtil.crashDumpsEnabled(false) or using system property -Dorg.deeplearning4j.crash.reporting.enabled=false
[main] WARN org.deeplearning4j.util.CrashReportingUtil - Memory crash dump reporting output location can be set with CrashUtil.crashDumpOutputDirectory(File) or using system property -Dorg.deeplearning4j.crash.reporting.directory=<path>
Exception in thread "main" java.lang.OutOfMemoryError: Failed to allocate memory within limits: totalBytes (140M + 32520M) > maxBytes (7262M)
34

A low utilization isn’t quite unexpected when you have a rather small model working on rather small data. But I would still expect a bit more than 1 to 2 percent on your GPU. How are you measuring that?

As for your crash… that is pretty weird. Especially because the crash dump you’ve shared previously happens on pointer deallocation and you’ve got plenty of free memory.

Just in case that you are hitting a bug that may have already been fixed, can you try using snapshots please? (See https://deeplearning4j.konduit.ai/config/config-snapshots)

35

Have the following issue with the latest snapshot:
I have also used Cuda 10.2. Can I then possibly also upgrade to 11.2?

Exception in thread "main" java.lang.NoClassDefFoundError: org/nd4j/common/config/ND4JClassLoading
	at org.nd4j.linalg.factory.Nd4jBackend.load(Nd4jBackend.java:137)
	at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5092)
	at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:270)
	at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:577)
Caused by: java.lang.ClassNotFoundException: org.nd4j.common.config.ND4JClassLoading
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 6 more
36

Yes, for snapshots you want to use 11.0 or 11.2

37

Which cuDNN version do I need for CUDA 11.2?

12:22:58.799 [main] INFO org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
12:23:00.379 [main] ERROR org.deeplearning4j.common.config.DL4JClassLoading - Cannot find class [org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper] of provided class-loader.
Exception in thread "main" java.lang.NullPointerException: Attempted to load class org.deeplearning4j.cuda.convolution.CudnnConvolutionHelper but failed. No class found with this name.
	at org.nd4j.common.base.Preconditions.throwNullPointerEx(Preconditions.java:643)
	at org.nd4j.common.base.Preconditions.checkNotNull(Preconditions.java:467)
	at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:100)
	at org.deeplearning4j.common.config.DL4JClassLoading.createNewInstance(DL4JClassLoading.java:89)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.initializeHelper(ConvolutionLayer.java:76)
38

@ForceUpdate1 we’re publishing docs on this for the release, I apologize for some of the side effects of being an early adopter :slight_smile:
Basically to use cudnn, there’s actually 2 ways:

  1. Use the new classifiers for nd4j found here:
    Index of /repositories/snapshots/org/nd4j/nd4j-cuda-11.2/1.0.0-SNAPSHOT
    (see anything with -cudnn in it)
  2. Use deeplearning4j-cuda. This is the old way. This can be found here:
    Index of /repositories/snapshots/org/deeplearning4j/deeplearning4j-cuda-11.0/1.0.0-SNAPSHOT
    and here:
    Index of /repositories/snapshots/org/deeplearning4j/deeplearning4j-cuda-11.2/1.0.0-SNAPSHOT

Now as for what you’re seeing, This is a bug that has been fixed recently. Please run mvn -U to ensure you get the updated dependencies. I would recommend trying #1 to see if that works for you.

39

It doesn’t look like it’s going to work. I once built the project with Maven “clean install -U” that should update the dependencies.
When I start the project from Intellij using Run, the same error occurs. Even after I have invalidated the cache.

If I start the output jar from maven this error comes:

13:50:25.205 [main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
13:50:25.208 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.nd4j.linalg.jblas.JblasBackend] of provided class-loader.
13:50:25.208 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.canova.api.io.data.DoubleWritable] of provided class-loader.
13:50:25.209 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.nd4j.linalg.jblas.JblasBackend] of provided class-loader.
13:50:25.209 [main] ERROR org.nd4j.common.config.ND4JClassLoading - Cannot find class [org.canova.api.io.data.DoubleWritable] of provided class-loader.
13:50:26.757 [main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 32
13:50:26.790 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Windows 10]
13:50:26.790 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [8]; Memory: [7,1GB];
13:50:26.790 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [CUBLAS]
13:50:26.795 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - ND4J CUDA build version: 11.2.142
13:50:26.796 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - CUDA device 0: [NVIDIA GeForce GTX 1070]; cc: [6.1]; Total memory: [8589934592]
13:50:26.797 [main] INFO org.nd4j.linalg.jcublas.JCublasBackend - Backend build information:
 MSVC: 192829914
STD version: 201402L
CUDA: 11.2.142
DEFAULT_ENGINE: samediff::ENGINE_CUDA
HAVE_FLATBUFFERS
13:50:26.830 [main] INFO org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
13:50:27.317 [main] ERROR org.nd4j.linalg.compression.BasicNDArrayCompressor - Error loading ND4J Compressors via service loader: No compressors were found. This usually occurs when running ND4J UI from an uber-jar, which was built incorrectly (without services resource files being included)
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.nd4j.linalg.factory.Nd4j.getCompressor(Nd4j.java:5314)
        at org.nd4j.linalg.api.ndarray.BaseNDArray.get(BaseNDArray.java:4122)
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:706)
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:604)
Caused by: java.lang.RuntimeException: Error loading ND4J Compressors via service loader: No compressors were found. This usually occurs when running ND4J UI from an uber-jar, which was built incorrectly (without services resource files being included)
        at org.nd4j.linalg.compression.BasicNDArrayCompressor.loadCompressors(BasicNDArrayCompressor.java:66)
        at org.nd4j.linalg.compression.BasicNDArrayCompressor.<init>(BasicNDArrayCompressor.java:46)
        at org.nd4j.linalg.compression.BasicNDArrayCompressor.<clinit>(BasicNDArrayCompressor.java:39)
        ... 6 more
40

@ForceUpdate1 that particular stack trace is completely unrelated to your problem. It’s complaining about some files that are missing. it actually looks like it loads the build fine. How are you building your jar exactly?

Edit: Just so you know what you’re looking at, nd4j leverages service providers for its various implementations of different interfaces like the compressor:

You can find those in every backend.

Here’s the cpu version:

I would encourage you to try again and ensure the right files are present, I highly doubt this is a bug (hence the very specific error message provided there)