Workspace for loading Google vectors model using loadStaticModel

Hi, people. I’m trying to load the google vectors model into DL4j, to use it later for sentiment analysis. I’m able to do it but with a big cost in RAM memory. I read about workspaces and how the memory is hold off-heap, so I tried to load it using loadStaticModel inside a workspace,just to see if my RAM consumption was lower. However, I’m not able to do it if I set my available amount of RAM to a number less of 3.4 GB (the size of Google vectors model). So, does it mean the workspaces are only for certain operations?. It’s that a bug? . the code is quite simple:

final WorkspaceConfiguration mmap = WorkspaceConfiguration.builder()
final File wordVectorsFile = new File("/sentiment/GoogleNews-vectors-negative3002.bin");
try (final MemoryWorkspace ws = Nd4j.getWorkspaceManager().getAndActivateWorkspace(mmap, "M2")) {
final WordVectors wordVectors = WordVectorSerializer.loadStaticModel(wordVectorsFile);

Any clues?.

Workspace in this case will be used for vectors only. But, obviously, won’t be used for, say, Strings in it. But ye, i’d expect it to work.

Yes, I want to use it to load the vectors file only, outside JVM memory, and use it later. However, I can’t allocate vectors so process for loading the model fails. It shouldn’t be allocated to the workspace instead of RAM?.

File an issue please, and I’ll take a look. I’d definitely expect it to work that way.


This is a different question but related to solve the issue: How do I know that my workspace is actually storing my array?

Take a look inside the file and see what is inside. It should be the binary representation of the contents of the arrays inside that workspace.

Oh, thanks. I did it and indeed, the workspace is used.However, is not used as much as I think it should be used, but that’s a different issue.