I am trying to figure out how to get a list of document ids that are false positives during evaluation. I have tried using CustomEvaluation but it seems that when calling EvaluationLambda, recordMetaData is null. I have stored the document id in recordMetaData in the DataSet object. It seems to me when calling net.doEvaluation(iterator, eval) in a ComputationGraph at least, the eval call does not pass the recordMetaData. I am going about this the wrong way?
You are right that
net.doEvaluation(...) doesn’t call the evaluator with metadata.
I’ve opened an issue to track the progress of changing this to an appropriate behavior: https://github.com/eclipse/deeplearning4j/issues/8718
In the meantime, you will have to workaround the issue by basically copy&pasting this part of the evaluation loop and passing in the metadata yourself:
You should also note that you have to actually enable your RecordReaderDataSetIterator to collect the metadata in the first place. You can either enable it on the Builder if you are using it, or use the setter after its construction.
Thanks for opening the issue. I will use the workaround for now.