Thanks. I am familiar with the concept of transitive dependencies.
The only reason I added those dependencies (datavec, core, datasets) was
that I was trying to duplicate the creation of the multi-step input
dataset, which I have on the desktop side, on the Android side. I’ll do
it another way which, in all probability may well be more memory
intensive. I’ll figfure it out.
Still, I would hope to reduce the size of the app. Adding dl4j support
increased it by more than a third. Not a huge issue but one that in the
long run may be a problem. This is why I asked about upcoming release
and the minimal backend.
@adonnini it’s not really about the upcoming release though. The main ask was no matter what version you use or what app you’re building just do some up front detective work with the gradle command I gave you so you know what’s being included in your app.
Apologies about deeplearing4j-core there. The UX there is mainly a legacy problem. core used to be a lot smaller and most people want computer vision. The getting started experience and some docs you might have read might have caused you to add those dependencies.
Yes, the question was asked before. The response was “do it yourself”.
My questions was simply a follow-up on something you said regarding an
upcoming “leaner back-end”. If the answer is, it’s not coming anytime
soon (i.e. it is not a priority), that’s fine.
@adonnini sorry I don’t quite understand… deeplearning4j-core is not a backend.
nd4j-native is. deeplearning4j-core is also not required.
Also…the opencv libraries and other things that are pulled in for every platform are not required either.
You just have to follow the same principal as nd4j-native and include only the platforms you need.
I’m sorry if I give off the impression that any of these are required. They’re not. That’s exactly what I was trying to communicate.
You literally just need deeplearning4j-nn and nd4j-native for most problems. All the other things you chose to include aren’t even useful for your time series problem.
I asked you to understand how to run the gradle dependency analysis so you could understand why your build file is so large.
I’d appreciate if you would try to understand it’s not a requirement that your build be 1.4g.
In the context of nd4j-native the main issue is openblas is pulled in which can be large. That’s what the newer nd4j-minimizer backend is supposed to address.
So yes that will be coming, yes openblas is a bit bigger but certainly not 1.4g.
Now if you came to me with the gradle dependency analysis and showed me you were doing it correctly with the bare minimum libraries and not pulling in deeplearning4j-core with the extra (unneeded) computer vision dependencies and you still had a problem then there might be something for me to do here. There’s just not though.
Asking you to configure your own build file and “do it yourself” is a reasonable expectation of any library user. You have full control over what you do and do not include in your project.
I think I may have inadvertently caused a few misunderstandings:
I appreciate your pointing out how I could use gradle tools to better
understand the dependencies used by my app. I used the tool you
suggested in the past. I just did not think of it when I added core,
datasets and datavec
You did not give me the impression that core, datasets and datavec
were required to run the network on the Android side. I never thought
that was the case. When I added core, datasets, and datavec I was
looking into creating the input dataset for the network in a way which
would not use Java arrays. This is not a requirement for my app at all.
I was just exploring a potential alternative to how I am creating input
datasets now.
I have taken the diy approach to building build.gradle and modifying
dependency libraries several times in the past. Sometimes, it’s the only
way. It’s not fun, and can be time consuming, at least for me.
I wasn’t really asking you to do anything other than give me an
update on the nd4j-minimizer backend. At present, the apk of my app is
around 470 MB, a bit big but manageable. Smaller would be better but it
is not critical. I am quite happy to wait until you release the
nd4j-minimizer backend.
@adonnini thanks for clarifying. Try to elaborate on some of the context from the experiments from now on please. That will help me understand what you are missing.
Random experimentation in general is a foreign concept to me. It’s just not how I write code. Being a library author myself I know not to fight the framework I’m using.
If something exists I try to understand why. Someone wrote that code with some sort of bias or intention in mind. If you fight that it just makes your own life harder.
Regarding nd4j-minimizer it’s basically done it’s just a matter of doing QA for the release.
Thanks. I appreciate the advice. I will do my best to be clearer from
now on. Reading messages a couple of times before sending them is a good
practice which sometimes I forget.
I hope you are well. I am hesitating to contact you because I have a
pretty big problem I am not sure how to diagnose let alone resolve.
I have been experimenting with different network configurations using
different layers and, among other parameters, regularization.
Everything was working normally, i.e. inference results from all
experiments made sense.
Then, starting this past Saturday, inference results stopped making
sense, i.e. they were completely off.
I made no changes to the code on the desktop side. Training of the
original 2-layer network (and of the others I have been experimenting
with) started yielding results that were very different from training
runs made before Saturday.
Attached you will find the screen shot from the latest training run of
the original 2-layer network. I am also sending config.json of the saved
network.
I updated the code on github.
I am at a total loss as to what is going on. It’s likely that,
inadvertently, I made a change somewhere. I just don’t know.
By the way, running networks saved from before Saturday works normally
producing results that make sense. This leads me to belive that on the
Android side the code is working correctly.
Any ideas, suggestions, thoughts would be greatly appreciated. Please
let me know if you need any additional information.
I resolved my problem. it had to do with the input dataset. Everything
is back to working as it was before.
However, this issue highlighted how I still don’t understand as well as
I should how the input data is processed inside the network and how it
affects inference results.
Any suggestions as to what I could do to gain a better understanding
would be greatly appreciated.
