Imported Pretrained Keras Model Produces Wrong Output in DL4J

I have a Keras 2.3.1 h5 CNN fully trained model that I import using DL4J 1.0.0-beta6 version. I intend to use DL4J for evaluating outputs for specific input images. Currently Keras prediction outputs don’t match DL4J outputs. The shapes of the inputs are different Keras input shape is (4, 150, 150, 3) whereas DL4J
input shape is [4, 3, 150, 150]. I could not find any sample DL4J code which
performs predictions on a CNN (or any network) trained model imported from
Keras that use images for predictions in Java and hence cannot ascertain if this
is an issue.

Questions:

  1. What is the right shape of input required ini DL4J for the imported Keras
    Model that expects image input.

  2. If input shapes have to match between DL4J and Keras, I cannot identify a
    way to build a “channel last” INDArray from a set of input images.

  3. I need to also perform this on Keras imported mixed multi-input and
    multi-output networks. What is recommended way to feed inputs to such networks
    for predictions.

CURRENT CODE SNIPPET:

Keras:

	...
	# Build and train the CNN Model
	...
	imgGenerator = ImageDataGenerator(rescale=1./255)
	testDir = "/test"

	testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
	testDf = pd.DataFrame(testData, columns=["file"])
	netInput=imgGenerator.flow_from_dataframe( dataframe=testDf, 
		directory=testDir, x_col="file", y_col=None, 
		batch_size=testData.shape[0], seed=42, shuffle=False, class_mode=None, 
		target_size=(150, 150))
	netOutput = model.predict(netInput);
	print("Prediction: ", netOutput)
	
	model.save("sample.h5");

Result:

	Prediction:  [[0.01729406]
	 [0.76219445]
	 [0.40130898]
	 [0.9373665 ]]

Shape of the batch generated by netInput is (4, 150, 150, 3)

DL4J:

	String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
        
	INDArray[] ret = new INDArray[testImgs.length];
	int count = 0;
	for (String testImg: testImgs)
	{
		File f = new File(mInputPath + "/test", testImg);
		NativeImageLoader loader = new NativeImageLoader(150, 150, 3);
		INDArray image = loader.asMatrix(f);
		DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
		scalar.transform(image);
		ret[count] = image;
		count++;
	}
	// Concatenate the array along dimension 0.
	INDArray netInput = Nd4j.concat(0, ret);
	INDArray netOutput = model.output(netInput);
	System.out.println("Prediction: ", netOutput)

Result:

	Prediction: [[0.0381], 
	 [0.7351], 
	 [0.5337], 
	 [0.8388]]

Shape of netInput is [4, 3, 150, 150]

edit: make code snippets readable

All DL4J CNN 2D layers - including those obtained from Keras import - expect NCHW format data. ImageRecordReader/NativeImageLoader produces NCHW data by default. So this should not be the issue. Can you upload the model?
There are MultiDataSetIterator that you can build and use for (3).

Here is the model generation and training

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss='binary_crossentropy',
	optimizer=optimizers.RMSprop(lr=1e-4),
	metrics=['acc'])

trainDataGenerator = ImageDataGenerator( rescale=1./255, rotation_range=40, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)

trainDir = baseDir + "/train"
trainMetaFile = trainDir + "/train.csv"
trainDf = pd.read_csv(trainMetaFile, dtype=str)
trainGenerator=trainDataGenerator.flow_from_dataframe( dataframe=trainDf, directory=trainDir, x_col="file", y_col="label", batch_size=20, seed=42, shuffle=True, classes=CLASSES, class_mode="binary", target_size=(150, 150))
        
validationDir = baseDir + "/validation"
validationMetaFile = validationDir + "/validation.csv"
validationDf = pd.read_csv(validationMetaFile, dtype=str)
validationGenerator=trainDataGenerator.flow_from_dataframe( dataframe=validationDf, directory=validationDir, x_col="file", y_col="label", batch_size=20, seed=42, shuffle=True, classes=["dog", "cat"], class_mode="binary", target_size=(150, 150))

model.fit_generator( trainGenerator, steps_per_epoch=100, epochs=100,
	validation_data=validationGenerator, validation_steps=50)

edit: make code snippets readable

I retrained the above model and retried - still same problem. Please find the trained model H5, test images and test programs below:

A] Model and data
Download the model from cnnSample1Model.h5 - Google Drive and save it as /tmp/cnnSample1Model.h5

Download and untar the test data images from testData.tar.gz - Google Drive under /tmp/testData

B] DL4J Test Program:


package com.sample;

import static edu.sciproj.libcore.CCommandLineOption.nonEmpty;

import java.io.File;
import java.io.IOException;
import java.util.Arrays;

import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.nn.modelimport.keras.KerasModelImport;
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler;
import org.nd4j.linalg.factory.Nd4j;

// MultiDataSet handles multiple inputs and outputs.
public class TestCnn
{
    private static final int[] TARGET_SIZE = new int[]{150, 150};   // w, h
    private static final int IMAGE_CHANNEL = 3;
    private String mInputPath;
    
    public TestCnn(String modelFile, String inputDir)
    {
        mInputPath = inputDir;
    }
    
    public INDArray getInput() throws IOException
    {
        String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
        
        INDArray[] ret = new INDArray[testImgs.length];
        int count = 0;
        for (String testImg: testImgs)
        {
            File f = new File(mInputPath, testImg);
            NativeImageLoader loader = new NativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL);
            INDArray image = loader.asMatrix(f);
            DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
            scalar.transform(image);
            ret[count] = image;
            count++;
        }
        // Concatenate the array along dimension 0.
        return(Nd4j.concat(0, ret));
    }
    
    public MultiLayerNetwork loadModel(String modelFile)
        throws IOException, 
        InvalidKerasConfigurationException,
        UnsupportedKerasConfigurationException
    {
        return(KerasModelImport.importKerasSequentialModelAndWeights(modelFile));
    }
    
    public static void main(String[] args) 
        throws Exception
    {
        String homeDir = System.getenv("HOME");
        String modelFile = "/tmp/cnnSample1Model.h5";
        String inputDir = "/tmp/testData";
        
        TestCnn loader = new TestCnn(modelFile, inputDir);
        loader.getInput();
        MultiLayerNetwork model = loader.loadModel(modelFile);
        INDArray features = loader.getInput();
        long[] shape = features.shape();
        System.out.println("Feature shape=" + Arrays.toString(shape));
        INDArray output = model.output(features);
        
        System.out.println("Prediction: " + output);
    }
}

Output:
Prediction: [[0.0426],
[0.8121],
[0.7653],
[0.8947]]


C] Keras Test Program

import pandas as pd

import numpy as np
import sys
import traceback
from keras.models import load_model
from keras.preprocessing.image import ImageDataGenerator

def main(argv):
    try:
        model = load_model("/tmp/cnnSample1Model.h5")
        
        imgGenerator = ImageDataGenerator(rescale=1./255)
        testDir = "/tmp/testData"
        
        testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
        testDf = pd.DataFrame(testData, columns=["file"])
        itr=imgGenerator.flow_from_dataframe(dataframe=testDf, directory=testDir,
            x_col="file", y_col=None, batch_size=testData.shape[0], seed=42,
            shuffle=False, class_mode=None, target_size=(150, 150))
        
        netOutput = model.predict(itr);
        print("Prediction: ", netOutput)
        
    except Exception as excp:
        print("Failed processing")
        print(excp)
        traceback.print_exc()
        sys.exit(1)
    
if __name__ == '__main__':
    main(sys.argv)      

Prediction: [[0.01988904]
[0.8130257 ]
[0.7170239 ]
[0.9342622 ]]

As you can see, the DL4J and Keras outputs are not identical for same test
images. Please help.

edit: make code snippets readable

Everything points to a difference between RBG and BGR.
NaviveImageLoader loads images in the format that OpenCV loads them: BGR.

Unfortunately, there is no direct option to tell it to reverse this. However, Java2DNativeImageLoader does support it.

By modifiying your code to use it and request channel flipping like this, we get a result that is closer to what you have shown.

Java2DNativeImageLoader javaLoader = new Java2DNativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL);
INDArray image = javaLoader.asMatrix(ImageIO.read(f), true);
Prediction: [[0.0194], 
 [0.8040], 
 [0.7122], 
 [0.9308]]

Edit:

As I’ve just learned, there is actually a simple way to get the RGB images from NativeImageLoader:

NativeImageLoader javaLoader = new NativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL,  new ColorConversionTransform(COLOR_BGR2RGB));

By passing the ColorConversionTransform like that, it will transform the channels into the order that your network expects them.

There are many other Transforms, that would also support transformations to other schemes like HSV, HSL or Luv.