Building package for a different OS

DL4J
1

My development computer is a MacBook Pro, and I am using it along with DL4J to build deep learning based User Defined Functions for a Big Data Management System deployed on a cluster of linux machines with NVIDIA GPU’s.

Because I am building on my MacBook Pro, Maven tries to pull the macosx-x86_64-version of cuda-10.1, which I believe doesn’t exist as NVIDIA has dropped support for OSX, rendering me with the following error:

[ERROR] Failure to find org.nd4j:nd4j-cuda-10.1:jar:macosx-x86_64:1.0.0-beta6 in Central Repository: was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced → [Help 1]

So as of now my current workflow is to code on my MacBook Pro, push to a repo, pull it on one of the servers on the cluster, run mvn package there (where it builds without error), then deploy it.

Would it be possible to build it directly for a linux machine on my MacBook Pro?

2

You can bundle both nd4j-native-platform and nd4j-cuda-x.x-platform, so in runtime we’ll pick appropriate backend. On your mbp it’ll be nd4j-native, and on server it’ll be nd4j-cuda

3

Cool.

Out of curiosity how does it pick the appropriate backend when they are all added as dependencies?

4

CUDA is considered to have higher priority, so if GPU is available - it’ll be loaded first. If it’s not available, initialization will fail and nd4j-native will be loaded after that.

2 Likes
5

There are also environment variables for this, from https://nd4j.org/backend:

For enabling different backends at runtime, you set the priority with your environment via the environment variable:

BACKEND_PRIORITY_CPU=SOME_NUM
BACKEND_PRIORITY_GPU=SOME_NUM

Relative to the priority, it will allow you to dynamically set the backend type. More can be found here

1 Like