Hi. I’m developping an application using DL4J with hadoop. i added “deeplearning4j-hadoop” dependancy in my pom.xml. I got an error saying that the depandancy is not found in maven repository. I thought the problem came from inteliJ IDE, i switch to eclipse but the problem persist. someone have an idea what is going wrong?
Thank in advance.
The last release of the deeplearning4j-hadoop package was in 2014. Afterwards it was deprecated.
Thank you for answer
Is there a way to use deeplearning4j with hadoop?
It depends on what you want to do, in principle you can use it like any other java library with no need for a special integration.
I’m working on a project that aim is to train a model on a distributed model like hadoop. I already have hadoop cluster,now i willing to run my model on the cluster
The distributed training support was removed from dl4j a few releases ago, because it was unmaintained and tended to work only if you had a very specific setup.
If you know what you are doing, and understand what the phrases model parallel and data parallel mean, it is still possible to do so, but you’d need to implement parts of it yourself.
I’m start learning model parallel, i’m not an expert. It there another library i can use to have my work done?
@dibeb paul answered your question but I figured I would chime in for future users.
Model parallelism typically requires work done at the c++ level for speed. Multi gpu sharding is implemented in a few libraries out there. I’ll be working on incorporating kernels from some of those that should allow for better support there.