Loading model takes over two hours?!

The code you’ve shared in your original post uses reflection, i.e. getDeclaredMethod and invoke.

As we can’t quite trust your pom.xml file here, you can check it at runtime with:

        System.out.println(VersionCheck.versionInfoString());

That will print all versions for you.

I’ve downloaded the dutch w2v compatible file from the repository you’ve linked (http://vectors.nlpl.eu/repository/20/39.zip).

As I’m running version 1.0.0-beta7, I applied the workaround from the linked post (Beta 7 - Glove Word Vector - #4 by treo), loaded the txt file and stopped the time:

Nd4j.getEnvironment().allowHelpers(false);
long start = System.nanoTime();
Word2Vec word2vec = WordVectorSerializer.readWord2VecModel(new File("C:\\Users\\dubs\\Downloads\\39\\model.txt"));
long stop = System.nanoTime();
System.out.println("runtime = " + (stop - start) / 1e9);

This tells me that it took 82 seconds to load.

I repeated this with the binary model file (model.bin) and without the workaround, as that isn’t needed for binary model loading:

long start = System.nanoTime();
Word2Vec word2vec = WordVectorSerializer.readWord2VecModel(new File("C:\\Users\\dubs\\Downloads\\39\\model.bin"));
long stop = System.nanoTime();
System.out.println("runtime = " + (stop - start) / 1e9);

This took 57 seconds to load.

1 Like