Error: _reductionPointer allocation failed when loading model

Getting this error when trying to load my model, any idea what could be going wrong?

Caused by: java.lang.RuntimeException: _reductionPointer allocation failed; Error code: [2]
at org.nd4j.nativeblas.Nd4jCuda.lcScalarPointer(Native Method) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.jita.handler.impl.CudaZeroHandler.getCudaContext(CudaZeroHandler.java:1210) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.jita.handler.impl.CudaZeroHandler.alloc(CudaZeroHandler.java:225) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:526) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:438) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.initPointers(BaseCudaDataBuffer.java:297) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.<init>(BaseCudaDataBuffer.java:316) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.jcublas.buffer.CudaLongDataBuffer.<init>(CudaLongDataBuffer.java:87) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.jcublas.buffer.factory.CudaDataBufferFactory.createLong(CudaDataBufferFactory.java:1119) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.jcublas.buffer.factory.CudaDataBufferFactory.createLong(CudaDataBufferFactory.java:1114) ~[nd4j-cuda-10.1-1.0.0-beta6.jar:?]
at org.nd4j.linalg.factory.Nd4j.createBufferDetachedImpl(Nd4j.java:1296) ~[nd4j-api-1.0.0-beta6.jar:1.0.0-beta6]
at org.nd4j.linalg.factory.Nd4j.createBufferDetached(Nd4j.java:1278) ~[nd4j-api-1.0.0-beta6.jar:1.0.0-beta6]
at org.nd4j.linalg.factory.Nd4j.read(Nd4j.java:2552) ~[nd4j-api-1.0.0-beta6.jar:1.0.0-beta6]
at org.deeplearning4j.util.ModelSerializer.restoreMultiLayerNetwork(ModelSerializer.java:256) ~[deeplearning4j-nn-1.0.0-beta6.jar:?]
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.load(MultiLayerNetwork.java:3822) ~[deeplearning4j-nn-1.0.0-beta6.jar:?]

The dependencies in my pom.xml look like this

<dependencies>

    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-native-platform</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-native</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-api</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-core</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-nlp-uima</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>${nd4j.backend}</artifactId>
        <version>${nd4j.version}</version>
    </dependency>

    <!-- Core DL4J functionality -->
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-nlp</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-zoo</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <!-- ParallelWrapper & ParallelInference live here -->
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-parallel-wrapper</artifactId>
        <version>${dl4j.version}</version>
    </dependency>

    <!-- CSV-reader for the training data -->
    <dependency>
        <groupId>com.univocity</groupId>
        <artifactId>univocity-parsers</artifactId>
        <version>2.8.3</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.bytedeco/javacpp -->
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp</artifactId>
        <version>1.5.2</version>
    </dependency>

    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>nd4j-native-api</artifactId>
        <version>${nd4j.version}</version>
    </dependency>

</dependencies>

Funny thing is, it only happens on occacion, but like 1/3 of the time the code works as expected.

At time it happens when converting to a parallel inference model too.

Exception in thread “InferenceThread-1” java.lang.RuntimeException: java.lang.RuntimeException: _reductionPointer allocation failed; Error code: [2]
at org.deeplearning4j.parallelism.ParallelInference$InferenceWorker.run(ParallelInference.java:562)
Caused by: java.lang.RuntimeException: _reductionPointer allocation failed; Error code: [2]
at org.nd4j.nativeblas.Nd4jCuda.lcScalarPointer(Native Method)
at org.nd4j.jita.handler.impl.CudaZeroHandler.getCudaContext(CudaZeroHandler.java:1210)
at org.nd4j.jita.handler.impl.CudaZeroHandler.alloc(CudaZeroHandler.java:225)
at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:526)
at org.nd4j.jita.allocator.impl.AtomicAllocator.allocateMemory(AtomicAllocator.java:438)
at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.initPointers(BaseCudaDataBuffer.java:297)
at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.(BaseCudaDataBuffer.java:316)
at org.nd4j.linalg.jcublas.buffer.CudaFloatDataBuffer.(CudaFloatDataBuffer.java:63)
at org.nd4j.linalg.jcublas.buffer.factory.CudaDataBufferFactory.create(CudaDataBufferFactory.java:380)
at org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:1455)
at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.create(JCublasNDArrayFactory.java:1542)
at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.create(JCublasNDArrayFactory.java:1537)
at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4274)
at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:3962)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:691)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:607)
at org.deeplearning4j.parallelism.ParallelInference$InferenceWorker.initializeReplicaModel(ParallelInference.java:480)
at org.deeplearning4j.parallelism.ParallelInference$InferenceWorker.run(ParallelInference.java:498)

That’s OOM. I’d suggest to try latest snapshots.

For which packages should I add the latest-tag to the maven versioning?

You’ve been using beta6 back when you originally posted the question, since then beta7 has been released, so you can try that before going straight to snapshots.