libnd4jcpu.dylib+0x521c574] samediff:
:acquiredThreads(unsigned int)+0x4
I intemittently get this problematic frame as the error with no other real information on whats caused the error. I am not using SameDiff directly.
Anyone any ideas what this is pointing to?
Many thanks.
Hi @Cpt_Whittmann could you report more? Sorry this wasn’t really actionable.
We’d need more input. Was there a particular method call that caused this?
Hi,
I have setup a toy example with known regression approximation results that the network should produce quite easily. Mostly when i fit the network everything runs absolutely fine without any problems. Just occasionally my network runs into this same native error(and other associated ones) when fitting. Even if I change nothing and run the fit again quite often things work no problem.
I have been using the EarlyStopping mechanism and I have now taken this out as I have had problems with it regards scheduled learning rates. Ill test to see if this maybe the cause of the error.
Below are some error log files (stacktrace only):
1.
C [libnd4jcpu.dylib+0x521c574] _ZN8samediff6Ticket15acquiredThreadsEj+0x4
C [libnd4jcpu.dylib+0x5218064] _ZN8samediff10ThreadPool10tryAcquireEi+0xb4
C [libnd4jcpu.dylib+0x521ab69] _ZN8samediff7Threads11parallel_doENSt3__18functionIFvyyEEEy+0x39
C [libnd4jcpu.dylib+0xf2a8] ZN19NativeOpExecutioner19execTransformStrictEPN2sd13LaunchContextEiPKvPKxS4_S6_PvS6_S7_S6_S7_S6_S6+0x198
C [libnd4jcpu.dylib+0x2a7d9] execTransformStrict+0x99
C [libjnind4jcpu.dylib+0x11a47e] Java_org_nd4j_nativeblas_Nd4jCpu_execTransformStrict__Lorg_bytedeco_javacpp_PointerPointer_2ILorg_nd4j_nativeblas_OpaqueDataBuffer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_nd4j_nativeblas_OpaqueDataBuffer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_Pointer_2+0x36e
j org.nd4j.nativeblas.Nd4jCpu.execTransformStrict(Lorg/bytedeco/javacpp/PointerPointer;ILorg/nd4j/nativeblas/OpaqueDataBuffer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/LongPointer;Lorg/nd4j/nativeblas/OpaqueDataBuffer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/Pointer;)V+0
J 4334 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/TransformOp;Lorg/nd4j/linalg/api/ops/OpContext;)V (1440 bytes) @ 0x000000010fe6befc [0x000000010fe5e060+0x000000000000de9c]
J 4331 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/Op;Lorg/nd4j/linalg/api/ops/OpContext;)Lorg/nd4j/linalg/api/ndarray/INDArray; (143 bytes) @ 0x000000010fe5a95c [0x000000010fe59860+0x00000000000010fc]
J 4330 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/Op;)Lorg/nd4j/linalg/api/ndarray/INDArray; (7 bytes) @ 0x000000010fe593dc [0x000000010fe59340+0x000000000000009c]
j org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.execAndReturn(Lorg/nd4j/linalg/api/ops/TransformOp;)Lorg/nd4j/linalg/api/ops/TransformOp;+2
j org.nd4j.linalg.activations.impl.ActivationTanH.getActivation(Lorg/nd4j/linalg/api/ndarray/INDArray;Z)Lorg/nd4j/linalg/api/ndarray/INDArray;+11
j org.deeplearning4j.nn.layers.BaseLayer.activate(ZLorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/linalg/api/ndarray/INDArray;+16
j org.deeplearning4j.nn.layers.AbstractLayer.activate(Lorg/nd4j/linalg/api/ndarray/INDArray;ZLorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/linalg/api/ndarray/INDArray;+9
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.ffToLayerActivationsInWs(ILorg/deeplearning4j/nn/api/FwdPassType;ZLorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)Ljava/util/List;+383
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore()V+238
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)V+1
j org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/common/primitives/Pair;+13
j org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Z+85
j org.deeplearning4j.optimize.Solver.optimize(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)V+9
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)V+287
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)V+6
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(Lorg/nd4j/linalg/dataset/api/DataSet;)V+25
j org.deeplearning4j.parallelism.trainer.DefaultTrainer.fit(Lorg/nd4j/linalg/dataset/api/DataSet;)V+64
j org.deeplearning4j.parallelism.trainer.DefaultTrainer.run()V+643
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@13.0.1
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@13.0.1
j org.deeplearning4j.parallelism.ParallelWrapper$2$1.run()V+28
j java.lang.Thread.run()V+11 java.base@13.0.1
2.
C [libnd4jcpu.dylib+0x521c574] samediff::acquiredThreads(unsigned int)+0x4
C [libnd4jcpu.dylib+0x5218064] samediff::ThreadPool::tryAcquire(int)+0xb4
C [libnd4jcpu.dylib+0x521ab69] samediff::Threads::parallel_do(std::__1::function<void (unsigned long long, unsigned long long)>, unsigned long long)+0x39
C [libnd4jcpu.dylib+0xec28] NativeOpExecutioner::execTransformAny(sd::LaunchContext*, int, void const*, long long const*, void const*, long long const*, void*, long long const*, void*, long long const*, void*, long long const*, long long const*, bool)+0x258
C [libnd4jcpu.dylib+0x2a6b1] execTransformAny+0xa1
C [libjnind4jcpu.dylib+0x11966e] Java_org_nd4j_nativeblas_Nd4jCpu_execTransformAny__Lorg_bytedeco_javacpp_PointerPointer_2ILorg_nd4j_nativeblas_OpaqueDataBuffer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_nd4j_nativeblas_OpaqueDataBuffer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_Pointer_2+0x36e
j org.nd4j.nativeblas.Nd4jCpu.execTransformAny(Lorg/bytedeco/javacpp/PointerPointer;ILorg/nd4j/nativeblas/OpaqueDataBuffer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/LongPointer;Lorg/nd4j/nativeblas/OpaqueDataBuffer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/LongPointer;Lorg/bytedeco/javacpp/Pointer;)V+0
J 4097 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/TransformOp;Lorg/nd4j/linalg/api/ops/OpContext;)V (1440 bytes) @ 0x000000010e451aec [0x000000010e44d960+0x000000000000418c]
J 3876 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/Op;Lorg/nd4j/linalg/api/ops/OpContext;)Lorg/nd4j/linalg/api/ndarray/INDArray; (143 bytes) @ 0x000000010e407d7c [0x000000010e4077e0+0x000000000000059c]
J 3875 c1 org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(Lorg/nd4j/linalg/api/ops/Op;)Lorg/nd4j/linalg/api/ndarray/INDArray; (7 bytes) @ 0x000000010e4073c4 [0x000000010e407340+0x0000000000000084]
J 4092 c1 org.nd4j.linalg.api.ndarray.BaseNDArray.assign(Lorg/nd4j/linalg/api/ndarray/INDArray;)Lorg/nd4j/linalg/api/ndarray/INDArray; (104 bytes) @ 0x000000010e44aaac [0x000000010e44a3e0+0x00000000000006cc]
j org.nd4j.linalg.api.ndarray.BaseNDArray.mmuli(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)Lorg/nd4j/linalg/api/ndarray/INDArray;+475
j org.deeplearning4j.nn.layers.BaseLayer.preOutputWithPreNorm(ZZLorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/common/primitives/Pair;+305
j org.deeplearning4j.nn.layers.BaseLayer.preOutput(ZLorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/linalg/api/ndarray/INDArray;+4
j org.deeplearning4j.nn.layers.BaseOutputLayer.preOutput2d(ZLorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/linalg/api/ndarray/INDArray;+3
j org.deeplearning4j.nn.layers.BaseOutputLayer.backpropGradient(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/common/primitives/Pair;+9
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.calcBackpropGradients(Lorg/nd4j/linalg/api/ndarray/INDArray;ZZZ)Lorg/nd4j/common/primitives/Pair;+708
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore()V+384
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)V+1
j org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Lorg/nd4j/common/primitives/Pair;+13
j org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)Z+85
j org.deeplearning4j.optimize.Solver.optimize(Lorg/deeplearning4j/nn/workspace/LayerWorkspaceMgr;)V+9
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fitHelper(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)V+287
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;Lorg/nd4j/linalg/api/ndarray/INDArray;)V+6
j org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(Lorg/nd4j/linalg/dataset/api/DataSet;)V+25
j org.deeplearning4j.parallelism.trainer.DefaultTrainer.fit(Lorg/nd4j/linalg/dataset/api/DataSet;)V+64
j org.deeplearning4j.parallelism.trainer.DefaultTrainer.run()V+643
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@13.0.1
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@13.0.1
j org.deeplearning4j.parallelism.ParallelWrapper$2$1.run()V+28
j java.lang.Thread.run()V+11 java.base@13.0.1
Sorry, still need more than this. Would you mind giving me a code snippet that reproduces this? Thanks!