Hi all!
I’m new on deeplearning4j and I’m exploring the project.
Does Deeplearning4j provide a stand-alone inference engine as OpenVINO or Nvidia Triton?
The idea is to configure the inference engine withta pretrained model, developed with different framewors (tensorflow, keras, …), provide the data to the engine and get the output.
Does exist such a feature? If so, is there any documentation?
Thanks!
@pierantoniomerlino we have pretty good coverage for tensorflow and (in snapshots) onnx. We also have fairly good keras coverage. The only gotcha is importing keras uses an older api.
On various devices such as ARM, NVIDIA, Intel we also have platform specific bindings for cnns, rnns, matrix multiply like what you would also expect. If you can elaborate a bit more (maybe in DMs if it’s sensitive) I’d be happy to help direct you towards the right next steps.
@agibsonccc thank you for the reply. I’m a committer of Eclipse Kura, a open source IoT Edge Framework based on Java.
I’m exploring the possibilty to use deeplearning4j for adding some AI features to Kura. Our idea is to integrate an inference engine on Kura, so a user can upload their models and use the Kura features (as Wires, Driver/Asset, etc.) to get data from the field and feed the models. Then the result can be published on a cloud platform (i.e. Eclipse Kapua) or used for further analysis.
IMHO, the user should not develop or train the models on Kura, but only use it for inference.
@pierantoniomerlino great! Yeah this should be a great fit. I think we can fit your use case here, the main thing to be aware of are the different jars that bundle different pre compiled binaries for native platforms.
@agibsonccc Thanks for the reply! Have you some documentation or examples on the inference engine, so I can start playing with?
@pierantoniomerlino There’s a few parts I mentioned:
I’m happy to point you at whatever you would like, but I’d like to make sure I don’t send you on a wild goose chase.