Getting a list of false positive during evalution

mdavis95 · February 14, 2020, 10:13pm

I am trying to figure out how to get a list of document ids that are false positives during evaluation. I have tried using CustomEvaluation but it seems that when calling EvaluationLambda, recordMetaData is null. I have stored the document id in recordMetaData in the DataSet object. It seems to me when calling net.doEvaluation(iterator, eval) in a ComputationGraph at least, the eval call does not pass the recordMetaData. I am going about this the wrong way?

treo · February 18, 2020, 4:46pm

You are right that net.doEvaluation(...) doesn’t call the evaluator with metadata.

I’ve opened an issue to track the progress of changing this to an appropriate behavior: model.doEvaluation does not pass metadata · Issue #8718 · eclipse/deeplearning4j · GitHub

In the meantime, you will have to workaround the issue by basically copy&pasting this part of the evaluation loop and passing in the metadata yourself:

github.com

eclipse/deeplearning4j/blob/master/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/multilayer/MultiLayerNetwork.java#L3417-L3436


      
                  OutputLayerUtil.validateOutputLayerForClassifierEvaluation(outputLayer.conf().getLayer(), ROC.class);
              }
              return (T)doEvaluation(iterator, new org.deeplearning4j.eval.ROC(rocThresholdSteps))[0];
          }
          
          
/**
           * @deprecated To be removed - use {@link #evaluateROCMultiClass(DataSetIterator, int)} to enforce selection of appropriate ROC/threshold configuration
           */
          @Deprecated
          public <T extends ROCMultiClass> T evaluateROCMultiClass(DataSetIterator iterator) {
              return evaluateROCMultiClass(iterator, 0);
          }
          
          
/**
           * Evaluate the network on the specified data, using the {@link ROCMultiClass} class
           *
           * @param iterator          Data to evaluate on
           * @param rocThresholdSteps Number of threshold steps to use with {@link ROCMultiClass}
           * @return Multi-class ROC evaluation on the given dataset
           */

You should also note that you have to actually enable your RecordReaderDataSetIterator to collect the metadata in the first place. You can either enable it on the Builder if you are using it, or use the setter after its construction.

mdavis95 · February 18, 2020, 5:02pm

Thanks for opening the issue. I will use the workaround for now.

Topic		Replies	Views
How can I get all the misclassification examples? DL4J	3	345	April 20, 2020
Dl4j Scala quickstart DL4J	7	577	March 13, 2020
ROCMultiClass error while running textCNN DL4J	3	449	April 23, 2020
MultiDataSet's getExampleMetaData() doesn't work ND4J	1	202	December 3, 2022
Test issue in semantic segmentation using DL4J DL4J	8	738	January 27, 2020

Getting a list of false positive during evalution

Related topics