Limit native dependencies to one platform on SBT

DL4J
1

Hi,

Im aware of the recent similar solved/closed Limit native dependencies to one platform - #3 by sogawa-sps - but for me its about SBT and I have a open Stackoverflow issue on that here:

In short: My sbt universal:packageBin also packs up all platform dependent jars instead of the few Im interested in and on SBT I cant exclude them.
I also followed the hint to NOT depend on “nd4j-native-platform-1.0.0-M2”, but e.g “nd4j-native-1.0.0-M1-linux-x86_64” … but that saves only the nd4j-native jars. The other ones: javacpp-platform , hdf5-platform , openblas-platform , opencv-platform I still get pulled in in all variants through transitive dependencies that I dont control.

I can re-post the full stackoverflow issue here, if thats preferred.

Thanks for any help

1 Like
2

@jdo could you try setting javacpp.platform via a system property in sbt like: scala - How do I set a system property for my project in sbt? - Stack Overflow

Sorry I’m not as familiar with SBT’s internals but I do believe there’s a way to do it.

3

Hi @agibsonccc , thanks for the reply. I added

javacOptions ++= Seq("-J-Djavacpp.platform=windows-x86_64")

into the build.sbt but it doesnt do any difference and the fat-jar outcome still contains all platforms. To my understanding (from GitHub - bytedeco/javacpp-presets: The missing Java distribution of native C++ libraries & Reducing the Number of Dependencies · bytedeco/javacpp-presets Wiki · GitHub) this is a maven only trick and I already tried before the hinted “SBT-JavaCV”.

Just for my understanding - from pure (SBT) dependency management perspective there is no way to exclude them? (So basically SBT is not to blame)
The way the artifacts are declared, they will pull in all those platforms and the only way to control this is not via common dependency management declarations, but the specific (maven-first) tool that all those “org.bytedeco.*” libraries share ?

PS: just for the dimensions: a sibling module (same PlayFramework base, actually more business logic inside) has ~ 150 MB of gziped fatjar size. This one with the deeplearning4j dependency and a more narrow functional scope comes in at 1.2 GB gziped. The immediate problem is actually that is breaks the build pipeline now because of memory constrains.

4

@jdo thanks for confirming. Try following the instructions here: GitHub - bytedeco/sbt-javacpp: Use JavaCPP and JavaCPP presets with ease. Base plugin for JavaCPP-related projects. It looks like the plugin should handle that for you. I think Sam gave you the right advice following this here: Reducing the Number of Dependencies · bytedeco/javacpp-presets Wiki · GitHub

5

I think I solved it for my specific case. I could not get the “SBT-JavaCV” plugin to work. In short, I managed via plain SBT and excluded first all "org.bytedeco.xyz-platform " dependencies from my direct dependency “org.deeplearning4j.deeplearning4j-nlp:1.0.0-M2” and in a second step added explicitly the platform specific variants back that I need (via “classifier”).

The details are in java - Avoid platform specific dependencies in SBT packageBin - Stack Overflow