Imported Pretrained Keras Model Produces Wrong Output in DL4J

I have a Keras 2.3.1 h5 CNN fully trained model that I import using DL4J 1.0.0-beta6 version. I intend to use DL4J for evaluating outputs for specific input images. Currently Keras prediction outputs don’t match DL4J outputs. The shapes of the inputs are different Keras input shape is (4, 150, 150, 3) whereas DL4J
input shape is [4, 3, 150, 150]. I could not find any sample DL4J code which
performs predictions on a CNN (or any network) trained model imported from
Keras that use images for predictions in Java and hence cannot ascertain if this
is an issue.

Questions:

  1. What is the right shape of input required ini DL4J for the imported Keras
    Model that expects image input.

  2. If input shapes have to match between DL4J and Keras, I cannot identify a
    way to build a “channel last” INDArray from a set of input images.

  3. I need to also perform this on Keras imported mixed multi-input and
    multi-output networks. What is recommended way to feed inputs to such networks
    for predictions.

CURRENT CODE SNIPPET:

Keras:

# Build and train the CNN Model

imgGenerator = ImageDataGenerator(rescale=1./255)
testDir = “/test”

testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
testDf = pd.DataFrame(testData, columns=["file"])
netInput=imgGenerator.flow_from_dataframe( dataframe=testDf, 
	directory=testDir, x_col="file", y_col=None, 
	batch_size=testData.shape[0], seed=42, shuffle=False, class_mode=None, 
	target_size=(150, 150))
netOutput = model.predict(netInput);
print("Prediction: ", netOutput)

model.save("sample.h5");

Result:
Prediction: [[0.01729406]
[0.76219445]
[0.40130898]
[0.9373665 ]]

Shape of the batch generated by netInput is (4, 150, 150, 3)

DL4J:

String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
    
INDArray[] ret = new INDArray[testImgs.length];
int count = 0;
for (String testImg: testImgs)
{
	File f = new File(mInputPath + "/test", testImg);
	NativeImageLoader loader = new NativeImageLoader(150, 150, 3);
	INDArray image = loader.asMatrix(f);
	DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
	scalar.transform(image);
	ret[count] = image;
	count++;
}
// Concatenate the array along dimension 0.
INDArray netInput = Nd4j.concat(0, ret);
INDArray netOutput = model.output(netInput);
System.out.println("Prediction: ", netOutput)

Result:
Prediction: [[0.0381],
[0.7351],
[0.5337],
[0.8388]]

Shape of netInput is [4, 3, 150, 150]

All DL4J CNN 2D layers - including those obtained from Keras import - expect NCHW format data. ImageRecordReader/NativeImageLoader produces NCHW data by default. So this should not be the issue. Can you upload the model?
There are MultiDataSetIterator that you can build and use for (3).

Here is the model generation and training

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation=‘relu’, input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation=‘relu’))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation=‘relu’))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation=‘relu’))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation=‘relu’))
model.add(layers.Dense(1, activation=‘sigmoid’))

model.summary()

model.compile(loss=‘binary_crossentropy’,
optimizer=optimizers.RMSprop(lr=1e-4),
metrics=[‘acc’])

trainDataGenerator = ImageDataGenerator( rescale=1./255, rotation_range=40, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)

trainDir = baseDir + “/train”
trainMetaFile = trainDir + “/train.csv”
trainDf = pd.read_csv(trainMetaFile, dtype=str)
trainGenerator=trainDataGenerator.flow_from_dataframe( dataframe=trainDf, directory=trainDir, x_col=“file”, y_col=“label”, batch_size=20, seed=42, shuffle=True, classes=CLASSES, class_mode=“binary”, target_size=(150, 150))

validationDir = baseDir + “/validation”
validationMetaFile = validationDir + “/validation.csv”
validationDf = pd.read_csv(validationMetaFile, dtype=str)
validationGenerator=trainDataGenerator.flow_from_dataframe( dataframe=validationDf, directory=validationDir, x_col=“file”, y_col=“label”, batch_size=20, seed=42, shuffle=True, classes=[“dog”, “cat”], class_mode=“binary”, target_size=(150, 150))

model.fit_generator( trainGenerator, steps_per_epoch=100, epochs=100,
validation_data=validationGenerator, validation_steps=50)

I retrained the above model and retried - still same problem. Please find the trained model H5, test images and test programs below:

A] Model and data
Download the model from https://drive.google.com/open?id=1m57YG1_FfNxIQq1v6zSOScwy9hlVBWzs and save it as /tmp/cnnSample1Model.h5

Download and untar the test data images from https://drive.google.com/open?id=1-9LTXFGyr5SMn4PkSpFZhdKaKorI8a7E under /tmp/testData

B] DL4J Test Program:


package com.sample;

import static edu.sciproj.libcore.CCommandLineOption.nonEmpty;

import java.io.File;
import java.io.IOException;
import java.util.Arrays;

import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.nn.modelimport.keras.KerasModelImport;
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler;
import org.nd4j.linalg.factory.Nd4j;

// MultiDataSet handles multiple inputs and outputs.
public class TestCnn
{
private static final int TARGET_SIZE = new int{150, 150}; // w, h
private static final int IMAGE_CHANNEL = 3;
private String mInputPath;

public TestCnn(String modelFile, String inputDir)
{
    mInputPath = inputDir;
}

public INDArray getInput() throws IOException
{
    String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
    
    INDArray[] ret = new INDArray[testImgs.length];
    int count = 0;
    for (String testImg: testImgs)
    {
        File f = new File(mInputPath, testImg);
        NativeImageLoader loader = new NativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL);
        INDArray image = loader.asMatrix(f);
        DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
        scalar.transform(image);
        ret[count] = image;
        count++;
    }
    // Concatenate the array along dimension 0.
    return(Nd4j.concat(0, ret));
}

public MultiLayerNetwork loadModel(String modelFile)
    throws IOException, 
    InvalidKerasConfigurationException,
    UnsupportedKerasConfigurationException
{
    return(KerasModelImport.importKerasSequentialModelAndWeights(modelFile));
}

public static void main(String[] args) 
    throws Exception
{
    String homeDir = System.getenv("HOME");
    String modelFile = "/tmp/cnnSample1Model.h5";
    String inputDir = "/tmp/testData";
    
    TestCnn loader = new TestCnn(modelFile, inputDir);
    loader.getInput();
    MultiLayerNetwork model = loader.loadModel(modelFile);
    INDArray features = loader.getInput();
    long[] shape = features.shape();
    System.out.println("Feature shape=" + Arrays.toString(shape));
    INDArray output = model.output(features);
    
    System.out.println("Prediction: " + output);
}

}


Output:
Prediction: [[0.0426],
[0.8121],
[0.7653],
[0.8947]]


C] Keras Test Program

import pandas as pd

import numpy as np
import sys
import traceback
from keras.models import load_model
from keras.preprocessing.image import ImageDataGenerator

def main(argv):
try:
model = load_model("/tmp/cnnSample1Model.h5")

    imgGenerator = ImageDataGenerator(rescale=1./255)
    testDir = "/tmp/testData"
    
    testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
    testDf = pd.DataFrame(testData, columns=["file"])
    itr=imgGenerator.flow_from_dataframe(dataframe=testDf, directory=testDir,
        x_col="file", y_col=None, batch_size=testData.shape[0], seed=42,
        shuffle=False, class_mode=None, target_size=(150, 150))
    
    netOutput = model.predict(itr);
    print("Prediction: ", netOutput)
    
except Exception as excp:
    print("Failed processing")
    print(excp)
    traceback.print_exc()
    sys.exit(1)

if name == ‘main’:
main(sys.argv)


Prediction: [[0.01988904]
[0.8130257 ]
[0.7170239 ]
[0.9342622 ]]

As you can see, the DL4J and Keras outputs are not identical for same test
images. Please help.