Is it possible to train Deeplearning4j models on Apache Beam, so that no only Spark Engine is used, but Flink or Google DataFlow?
It would be nice to train Deeplearning4j models on SaaS like Google DataFlow in Scala and with data extracted from BigQuery with Spotify Scio.
Best Regards
Artur
At the moment there is no special support of Beam and as far as I’m aware of there are no near term plans to implement it. Maybe @agibsonccc can comment on this, too.
@arturroo we looked at it a while back but beam didn’t have enough marketshare. Compatibility and maintaining 2 different versions of our spark integration was just too costly. We’d accept contributions if you’re up for it or know someone who would be, otherwise it’s mostly an ROI thing for us.