DL4J and SQLDB : could not create cache

No, it’s in the JAR files, but some build tools can remove resource files, so you need to make sure this doesn’t happen. ND4J will require its resources to function. You can’t expect something to work if you remove parts of it!

Of course and as you can see from the pom file ,I am not removing anything from the jar files. the mvn package generates the uber jar that I copy straight to ksqldb .no mocking around .
I will dig into maven to see if there is a scenario in which it removes resource files .

the file is definetely in the uber jar:

(base) araji@infra:~/demo/dev/araji/confluent-latest/confluent-5.5.0$ jar -tvf …/…/ksql/ar-udf/anomaly-detection-udf-2.0-jar-with-dependencies.jar |grep nd4j-native.properties
1816 Fri Dec 13 23:21:38 EST 2019 nd4j-native.properties

and its content

(base) araji@infra:~/demo/dev/araji/confluent-latest/confluent-5.5.0/tmp$ cat nd4j-native.properties

real.class.double = org.nd4j.linalg.cpu.NDArray
complex.class.double = org.nd4j.linalg.cpu.nativecpu.complex.ComplexNDArray
shapeinfoprovider = org.nd4j.linalg.cpu.nativecpu.DirectShapeInfoProvider
constantsprovider = org.nd4j.linalg.cpu.nativecpu.cache.ConstantBuffersCache
affinitymanager = org.nd4j.linalg.cpu.nativecpu.CpuAffinityManager
memorymanager = org.nd4j.linalg.cpu.nativecpu.CpuMemoryManager
dtype = float
complex.double.class = org.nd4j.linalg.cpu.nativecpu.complex.ComplexDouble
blas.ops = org.nd4j.linalg.cpu.nativecpu.BlasWrapper

native.ops= org.nd4j.nativeblas.Nd4jCpu
ndarrayfactory.class = org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory
ndarray.order = c
resourcemanager_state = false
databufferfactory = org.nd4j.linalg.api.buffer.factory.DefaultDataBufferFactory
workspacemanager = org.nd4j.linalg.cpu.nativecpu.workspace.CpuWorkspaceManager
alloc = javacpp
fft = org.nd4j.linalg.fft.DefaultFFTInstance
opexec= org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner
opexec.mode= native

What happens if you remove everything but ND4J from the pom.xml file, and run it outside ksqlDB?

will do that and update the thread .

Thanks for hanging in there with me. :pray:

just double checked that it all works fine outside of ksql .

Will try the ksql support to better understand the overall process , Thanks again for an outstanding support .

1 Like