Workspace "WS_LAYER_WORKING_MEM" for array type FF_WORKING_MEM is not open

Hello,

After some graph.fit(trainerIter) I’d like to do something like:

    while (testIter.hasNext()) {
      scoreSum += graph.score(testIter.next());
      n++;
    }

But I get the following exception:

org.nd4j.linalg.workspace.ND4JWorkspaceException: Workspace "WS_LAYER_WORKING_MEM" for array type FF_WORKING_MEM is not open
	at 
org.nd4j.linalg.workspace.BaseWorkspaceMgr.enforceExistsAndActive(BaseWorkspaceMgr.java:325)
	at org.nd4j.linalg.workspace.BaseWorkspaceMgr.dup(BaseWorkspaceMgr.java:276)
	at org.deeplearning4j.util.ConvolutionUtils.reshape4dTo2d(ConvolutionUtils.java:537)
	at org.deeplearning4j.util.ConvolutionUtils.reshape4dTo2d(ConvolutionUtils.java:524)
	at 
org.deeplearning4j.nn.layers.convolution.CnnLossLayer.computeScore(CnnLossLayer.java:201)
	at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.computeScore(LayerVertex.java:252)
	at org.deeplearning4j.nn.graph.ComputationGraph.scoreHelper(ComputationGraph.java:3076)
	at org.deeplearning4j.nn.graph.ComputationGraph.score(ComputationGraph.java:3022)
	at org.deeplearning4j.nn.graph.ComputationGraph.score(ComputationGraph.java:3010)

I have no idea of what can cause this exception. I do not deal excplicitely with workspaces in my code.

Any advice ?

Looks like that maybe some kind of bug because it doesn’t expect to be scored outside of training?

@agibsonccc you might want to look into this.

@HGuillemet would you mind sharing the model definition?

The model is a bit complex and the code difficult to extract .
I’m sending you the output of graph.summary() as a private message, in case it would help.

EDIT: No way to send you a private message. Maybe if you send me one I will be able to reply.

You could just put that in a gist: https://gist.github.com/

@HGuillemet could you clarify why you can’t send a private message I’ve received a ton on the site. Can you try sending it to me?
Beyond that…I don’t think it has anything to do with the net. The loss against batches can typically be aggregated and tracked elsewhere already. I’m happy to look at this if you want to file an issue.
For now, for each dataset you can just do something like:

DataSet d = new DataSet();
DataSet dataSetFromWorkspace = iter.next();
DataSet d = new DataSet(dataSetFromWorkspace.getFeatures().detach(),dataSetFromWorkspace.getLabels().leverage());

This uses the INDArray.leverage() to detact it from the workspace and gives you a dataset in a similar manner.

I get a message “Sorry, you cannot send a personal message to that user.”. Maybe I must gather some badges before gaining this right…

I’m using MultiDataSet but I guess I can adapt this code to MultiDataSet.
I just tried using graph.evaluate instead of graph.score with a custom IEvaluation more or less reimplementing my loss function score and I don’t get the error in this case.

I will do it if I manage to extract some self-contained test case.

New issue created with test case.