DL4J Facenetnn4small2 Face Recognition project

I am working on a face recognition project using DL4J Core 1.0.0-M2.1 Facenetnn4small2. Despite progress, I am facing accuracy challenges, and my knowledge in transfer learning and pretrained models such as Tensorflow facenet.h5 model and weights is limited.
Kindly look at the code I have provided and offer me your guidance on enhancing accuracy, especially given our need to train and verify on limited dataset just like face unlock does with smart phones.

I have provided code for training Facenetnn4small2 model (Part 1) and for running face recognition(Part 2).

Training Model Part 1

package digitalattendance;

import org.deeplearning4j.zoo.ZooModel;
import org.deeplearning4j.zoo.model.FaceNetNN4Small2;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Random;
import java.util.logging.Level;
import org.datavec.api.io.filters.BalancedPathFilter;
import org.datavec.api.io.labels.ParentPathLabelGenerator;
import org.datavec.api.split.FileSplit;
import org.datavec.api.split.InputSplit;
import org.datavec.image.loader.NativeImageLoader;
import org.datavec.image.recordreader.ImageRecordReader;
import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator;
import org.deeplearning4j.nn.conf.*;
import org.deeplearning4j.nn.graph.ComputationGraph;
import org.deeplearning4j.optimize.listeners.ScoreIterationListener;
import org.deeplearning4j.util.ModelSerializer;
import org.nd4j.evaluation.classification.Evaluation;
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;


  • @author AugustineBeilel
    public class FaceNetSerialiser {

    private static final Logger logger = LoggerFactory.getLogger(FaceNetSerialiser.class);
    private long seed = 1234;
    CacheMode cacheMode = CacheMode.NONE;
    static int nEpochs = 10, transformedDataEpochs = 5;
    Random randNumGen = new Random(seed);
    File mainPath = new File(“D:\2Citt\Parliament\Dataset”);
    //File CnnModelPath = new File(“C:\TrackSMART\FaceNetModel” + “.zip”);
    File CnnModelPath = new File(“D:\2Citt\Parliament” + “.zip”);

    int batchSize = 54; // Test batch size ( number of samples that will be propagated through the network in each iteration )54

    public void FaceNetNN4Small2() throws IOException {
    int height = 96;
    int width = 96;

     FileSplit filesInDir = new FileSplit(mainPath, NativeImageLoader.ALLOWED_FORMATS, randNumGen);
     int numClasses = filesInDir.getRootDir().listFiles(File::isDirectory).length; //This only works if your root is clean: only label subdirs.
     //creating Face Labels txt file
     logger.info("\n\n creating Face Labels.txt file");
     File file = new File("C:\\TrackSMART\\Face Labels" + ".txt");
     try ( PrintWriter writer = new PrintWriter(file)) {
         for (int i = 0; i < numClasses; i++) {
             String folderName = filesInDir.getRootDir().listFiles(File::isDirectory)[i].getName();
             writer.println(folderName + "," + i);
     } catch (FileNotFoundException e) {
     // vectorization of train data
     ParentPathLabelGenerator labelMaker = new ParentPathLabelGenerator();
     BalancedPathFilter pathFilter = new BalancedPathFilter(randNumGen, NativeImageLoader.ALLOWED_FORMATS, labelMaker);
     //Split the image files into train and test. Specify the train test split as 80%,20%
     InputSplit[] filesInDirSplit = filesInDir.sample(pathFilter, 80, 20);
     InputSplit trainData = filesInDirSplit[0];
     InputSplit testData = filesInDirSplit[1];  //The testData is never used in the example, commenting out.
     // vectorization of train data
     logger.info("\n\n vectorization of train data");
     ImageRecordReader trainRR = new ImageRecordReader(height, width, 3, labelMaker);
     try {
     } catch (IOException ex) {
         java.util.logging.Logger.getLogger(FaceNetSerialiser.class.getName()).log(Level.SEVERE, null, ex);
     DataSetIterator trainIter = new RecordReaderDataSetIterator(trainRR, batchSize, 1, numClasses);
     // pixel values from 0-255 to 0-1 (min-max scaling)
     logger.info("\n\nTrain Data Normalistion (pixel values from 0-255 to 0-1 (min-max scaling))");
     DataNormalization imageScaler = new ImagePreProcessingScaler(0, 1);
     // vectorization of test data
     logger.info("\n\n vectorization of test data");
     ImageRecordReader testRR = new ImageRecordReader(height, width, 3, labelMaker);
     try {
     } catch (IOException ex) {
         java.util.logging.Logger.getLogger(FaceNetSerialiser.class.getName()).log(Level.SEVERE, null, ex);
     DataSetIterator testIter = new RecordReaderDataSetIterator(testRR, batchSize, 1, numClasses);
     logger.info("\n\nTest Data Normalistion (pixel values from 0-255 to 0-1 (min-max scaling))");
     testIter.setPreProcessor(imageScaler); // same normalization for better results
     logger.info("Multi Layer Neural Network configuration and training...");

    ////Neural Network Architecture
    ZooModel zooModel = FaceNetNN4Small2.builder()
    // .seed(seed)
    ComputationGraph model = ((FaceNetNN4Small2) zooModel).init();
    model.setListeners(new ScoreIterationListener(10));
    logger.info(“\n\nTotal num of params: {}” + model.numParams());
    // evaluation while training (the score should go down)
    logger.info(“Training with original data”);
    for (int i = 0; i < nEpochs; i++) {
    logger.info(“\n\nCompleted epoch {} " + i);
    Evaluation eval = model.evaluate(testIter);
    logger.info(”\n\n" + eval.stats());
    try {
    ModelSerializer.writeModel(model, CnnModelPath, true);
    } catch (IOException ex) {
    java.util.logging.Logger.getLogger(FaceNetSerialiser.class.getName()).log(Level.SEVERE, null, ex);
    logger.info("\n\nMultiLayer Neural Network model has been saved in {} " + CnnModelPath.getPath());

    public static void main(String args) throws IOException {
    FaceNetSerialiser fc = new FaceNetSerialiser();

Running Face recognition Part 2

//Use the nativeImageLoader to convert to numerical matrix // Load the image and convert it to a 4D array
NativeImageLoader loader = new NativeImageLoader(height, width, channels);
INDArray testimage = loader.asMatrix(croppedImage);
DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
//then call that scalar on the image dataset
INDArray embeddings = null;
try {
// Pass the input through the model to get the predictions
embeddings = model.output(testimage);
} catch (RuntimeException e) {
// NotificationManager.getInstance().fireNotification(Notification.Type.INFO, Notification.Location.CENTER, “Stopped Reading from the model”);


    Map<Integer, String> labelMap = new HashMap<>();
    File file = new File("C:\\TrackSMART\\Face Labels.txt");
    if (file.exists()) {
        try ( BufferedReader br = new BufferedReader(new FileReader(file))) {
            String line;
            while ((line = br.readLine()) != null) {
                if (line.trim().isEmpty()) {
                    continue; // skip empty lines
                String[] parts = line.split(",");
                if (parts.length >= 2) {
                    labelMap.put(Integer.parseInt(parts[1].trim()), parts[0].trim());
                } else {
        } catch (IOException e) {
    } else {

    int maxIdx = -1;
    double maxVal3 = Double.MIN_VALUE;
    double threshold = 0.997;
    boolean validPrediction = false;

    if (embeddings != null) {
        for (int k = 0; k < embeddings[0].size(1); k++) {
            double val = embeddings[0].getDouble(0, k);
            if (val > threshold) {
                validPrediction = true;
                if (val > maxVal3) {
                    maxVal3 = val;
                    maxIdx = k;
                    threshold = val;
                    //   System.out.println("Threshold value of the predicted label: " + threshold);
    } else {

    if (validPrediction) {
        StudentNumber = labelMap.get(maxIdx);
    } else {
        // handle case where no valid predictions were found

@AUGUSTINE-BEILEL to start, TF import uses a different api called samediff. Could you clarify what “tensorflow” you mean? Do you mean a pb file or a keras h5?

“Thank you for your response, @agibsonccc . I appreciate your willingness to help. To clarify, the ‘tensorflow’ I mentioned refers to the pretrained model format. I’m using the DL4J Core 1.0.0-M2.1 Facenetnn4small2 and aiming to improve accuracy in my face recognition project. Currently, my understanding of transfer learning and pretrained models is limited. Specifically, I’m referring to the ‘facenet.h5’ model and weights. Any guidance even if out of scope of my question you can provide on improving accuracy and fine-tuning on a limited dataset even if it means using pb file in DL4J, would be highly appreciated.”

Thank you once again.

@AUGUSTINE-BEILEL no nothing is out of scope. I wanted to clarify because tensorflow keras which was originally keras (but ironically is now multi backend again) has its own hdf5 based file format which is what you’re referring to.

What you’re specifying is not actually the “tensorflow” file format that I was asking about. That one is a separate protobuf (.pb) based format. That is the one that uses the different api and is lower level.

The .h5 format predates the .pb one and that’s the one we first supported. Hence that using the dl4j api.

In that case, could you clarify what your issue with importing was? If you want to improve accuracy, focus on making sure that your normalization and pre processing are equivallent to whatever you’re doing in python. Can you provide the code I asked for in that case?

Hello @agibsonccc,

Thank you for your explanation. I’m exclusively working with DL4J’s FaceNetNN4Small2 architecture for face recognition in Java. While I lack pre-trained models for this architecture, I’ve been advised to explore transfer learning using a pre-trained ‘facenet.h5’ model from Python.

Could you provide guidance on effectively importing and fine-tuning the ‘facenet.h5’ model within DL4J for improved accuracy, considering my limited dataset? Your insights would be greatly appreciated.

Thank you again for your assistance.

@AUGUSTINE-BEILEL you can import a keras model using the keras model import api and then use transfer our transfer learning api to adapt your model. Note that when importing you’ll either have a functional model or a sequential. There are 2 separate apis for those. Computation Graph for functional and MultiLayerNetwork for Sequential.