Workspace for loading Google vectors model using loadStaticModel

DL4J
1

Hi, people. I’m trying to load the google vectors model into DL4j, to use it later for sentiment analysis. I’m able to do it but with a big cost in RAM memory. I read about workspaces and how the memory is hold off-heap, so I tried to load it using loadStaticModel inside a workspace,just to see if my RAM consumption was lower. However, I’m not able to do it if I set my available amount of RAM to a number less of 3.4 GB (the size of Google vectors model). So, does it mean the workspaces are only for certain operations?. It’s that a bug? . the code is quite simple:

final WorkspaceConfiguration mmap = WorkspaceConfiguration.builder()
.initialSize(9500000000L)
.policyLocation(LocationPolicy.MMAP)
.policyAllocation(AllocationPolicy.STRICT)
.policyReset(ResetPolicy.ENDOFBUFFER_REACHED)
.tempFilePath("/Users/Downloads/temporalFile.temp")
.build();
final File wordVectorsFile = new File("/sentiment/GoogleNews-vectors-negative3002.bin");
try (final MemoryWorkspace ws = Nd4j.getWorkspaceManager().getAndActivateWorkspace(mmap, "M2")) {
final WordVectors wordVectors = WordVectorSerializer.loadStaticModel(wordVectorsFile);
}

Any clues?.

2

Workspace in this case will be used for vectors only. But, obviously, won’t be used for, say, Strings in it. But ye, i’d expect it to work.

3

Yes, I want to use it to load the vectors file only, outside JVM memory, and use it later. However, I can’t allocate vectors so process for loading the model fails. It shouldn’t be allocated to the workspace instead of RAM?.

4

File an issue please, and I’ll take a look. I’d definitely expect it to work that way.

5

Done!.

6

This is a different question but related to solve the issue: How do I know that my workspace is actually storing my array?

7

Take a look inside the file and see what is inside. It should be the binary representation of the contents of the arrays inside that workspace.

8

Oh, thanks. I did it and indeed, the workspace is used.However, is not used as much as I think it should be used, but that’s a different issue.