Build Documentation out of date?

DL4J
21

Hi @treo I’m working on this with @zamlz. Thanks for all the help!

I’m wondering, where would we add custom build flags, such as -g ?

Also, how would we get our project to use this locally compiled version of nd4j as opposed to the one from maven central? Not sure what to do with our pom to make that happen.

22

If you switch your version to 1.0.0-SNAPSHOT, it’ll use the installed version instead of the Maven Central version (which is 1.0.0-beta6 only)
One thing to be aware of: if you have the snapshots repo set up in your project - which isn’t necessary I should add when doing a local build (as described here) - maven can decide to download and use the new snapshots instead of using the locally built ones, when it sees that the online version is newer. So don’t include the snapshots repo in your project, or periodically build locally to make sure that’s most recent, to avoid that happening.

23

If you want to add custom flags you can do that by setting the appropriate environment variables:

  • CMAKE_CXX_FLAGS_RELEASE when building in release mode
  • CMAKE_CXX_FLAGS_DEBUG when building in debug mode

If all you want is to add -g in order to build in debug mode, you just have to add -Dlibnd4j.build=debug to the maven build command.

24

Thanks Treo and AlexBlack! We greatly appreciate your help! I changed the versions in my pom file to 1.0.0-SNAPSHOT. For some reason, its also trying to find other artifacts that I don’t believe we built. It also appears like its trying to pull this from https://oss.sonatype.org/content/repositories/snapshots.

Here is the full error message. I feel like I’m suppose to be specifying which architecture in my pom file somehow.

[ERROR] Failed to execute goal on project SimpleDl4j: Could not resolve dependencies for project com.lts.simpledl4j:SimpleDl4j:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: org.nd4j:nd4j-native:jar:android-arm:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:android-arm64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:android-x86:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:android-x86_64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:ios-arm64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:ios-x86_64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:macosx-x86_64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:windows-x86_64:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:linux-ppc64le:1.0.0-SNAPSHOT, org.nd4j:nd4j-native:jar:linux-armhf:1.0.0-SNAPSHOT: Failure to find org.nd4j:nd4j-native:jar:android-arm:1.0.0-SNAPSHOT in https://oss.sonatype.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of sonatype-nexus-snapshots has elapsed or updates are forced

25

I guess you are using nd4j-native-platform? When building locally, you usually aren’t building for all possible platforms, but only for the one you have. For this reason you have to change to the nd4j-native dependency. It should pick up the correct one then.

26

Nice catch treo! Thanks!

27

I’m runnng our codebase now and I’m getting an error message. I don’t see any exception or traceback.

==29990==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

The code worked just fine before, perhaps there is something I didn’t do properly while building the code?

28

Looks like you compiled it with address sanitizer support? If you do so, you also have to load it.

See ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD. · Issue #796 · google/sanitizers · GitHub

29

Interesting, I’ll see what I can do. I don’t think I intentially did that but probably my system’s GCC did that on its own.

30

I tried linking the libasan.so library in LD_PRELOAD and was still getting some weird errirs. So alternatively I tried disabling the sanitize feature from GCC. While that went successfully, it seems like building libnd4j does requre gcc to have the -lasan flag.

[100%] Linking CXX shared library libnd4jcpu.so
/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lasan

I’m gonna put sanitizing back in gcc and see what I can do from there.

31

Hey guys, mind taking a look into this? I’m able to link the right libraries I think with the LD_PRELOAD but it seems like there is some issue in terms of memory usage. https://gist.github.com/zamlz/80bb11ab79b2dc0f0258862ec945dc63

This output kinda reminds me of valgrind output to be honest.

32

I guess you are using the libnd4j debug mode? when you enable that, it adds all of the following flags:
-g -O0 -fPIC -fmax-errors=2 -fsanitize=address

That is what we typically use for our own debugging and hunting for memory access problems.

If you want anything other than that (like no address sanitizer) you are free to set them using the environment variable as described above. Or you can just modify the CMakeLists.txt file.

I guess you expected that only -g will be added with the debug flag. I apologize for wording it unclearly.

33

Ye, just build in release mode, Java doesn’t play good with Asan anyway. Plus performance of debug-mode compilation is awful

34

Yeah seems like it works just fine in release mode! Thanks guys! I added my flags to the CXX flags variable you mentioned in the post above instead.