Loading model takes over two hours?!

DL4J
4

The code you’ve shared in your original post uses reflection, i.e. getDeclaredMethod and invoke.

As we can’t quite trust your pom.xml file here, you can check it at runtime with:

        System.out.println(VersionCheck.versionInfoString());

That will print all versions for you.

I’ve downloaded the dutch w2v compatible file from the repository you’ve linked (http://vectors.nlpl.eu/repository/20/39.zip).

As I’m running version 1.0.0-beta7, I applied the workaround from the linked post (Beta 7 - Glove Word Vector - #4 by treo), loaded the txt file and stopped the time:

Nd4j.getEnvironment().allowHelpers(false);
long start = System.nanoTime();
Word2Vec word2vec = WordVectorSerializer.readWord2VecModel(new File("C:\\Users\\dubs\\Downloads\\39\\model.txt"));
long stop = System.nanoTime();
System.out.println("runtime = " + (stop - start) / 1e9);

This tells me that it took 82 seconds to load.

I repeated this with the binary model file (model.bin) and without the workaround, as that isn’t needed for binary model loading:

long start = System.nanoTime();
Word2Vec word2vec = WordVectorSerializer.readWord2VecModel(new File("C:\\Users\\dubs\\Downloads\\39\\model.bin"));
long stop = System.nanoTime();
System.out.println("runtime = " + (stop - start) / 1e9);

This took 57 seconds to load.

1 Like