# Imported Pretrained Keras Model Produces Wrong Output in DL4J
**URL:** https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120
**Category:** DL4J
**Created:** [February 7, 2020, 8:49pm UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120 "2020-02-07T20:49:18Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@j1234d2003](https://community.konduit.ai/u/j1234d2003)
#### Post date: [February 7, 2020, 8:49pm UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120/1 "2020-02-07T20:49:18Z")
I have a Keras 2.3.1 h5 CNN fully trained model that I import using DL4J 1.0.0-beta6 version. I intend to use DL4J for evaluating outputs for specific input images. Currently Keras prediction outputs don’t match DL4J outputs. The shapes of the inputs are different Keras input shape is (4, 150, 150, 3) whereas DL4J
input shape is [4, 3, 150, 150]. I could not find any sample DL4J code which
performs predictions on a CNN (or any network) trained model imported from
Keras that use images for predictions in Java and hence cannot ascertain if this
is an issue.
Questions:
1. What is the right shape of input required ini DL4J for the imported Keras
Model that expects image input.
2. If input shapes have to match between DL4J and Keras, I cannot identify a
way to build a “channel last” INDArray from a set of input images.
3. I need to also perform this on Keras imported mixed multi-input and
multi-output networks. What is recommended way to feed inputs to such networks
for predictions.
CURRENT CODE SNIPPET:
Keras:
```python
...
# Build and train the CNN Model
...
imgGenerator = ImageDataGenerator(rescale=1./255)
testDir = "/test"
testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
testDf = pd.DataFrame(testData, columns=["file"])
netInput=imgGenerator.flow_from_dataframe( dataframe=testDf,
directory=testDir, x_col="file", y_col=None,
batch_size=testData.shape[0], seed=42, shuffle=False, class_mode=None,
target_size=(150, 150))
netOutput = model.predict(netInput);
print("Prediction: ", netOutput)
model.save("sample.h5");
```
Result:
```auto
Prediction: [[0.01729406]
[0.76219445]
[0.40130898]
[0.9373665]]
```
Shape of the batch generated by netInput is (4, 150, 150, 3)
DL4J:
```java
String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
INDArray[] ret = new INDArray[testImgs.length];
int count = 0;
for (String testImg: testImgs)
{
File f = new File(mInputPath + "/test", testImg);
NativeImageLoader loader = new NativeImageLoader(150, 150, 3);
INDArray image = loader.asMatrix(f);
DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
scalar.transform(image);
ret[count] = image;
count++;
}
// Concatenate the array along dimension 0.
INDArray netInput = Nd4j.concat(0, ret);
INDArray netOutput = model.output(netInput);
System.out.println("Prediction: ", netOutput)
```
Result:
```auto
Prediction: [[0.0381],
[0.7351],
[0.5337],
[0.8388]]
```
Shape of netInput is [4, 3, 150, 150]
edit: make code snippets readable
---
### Author:  [@eraly](https://community.konduit.ai/u/eraly)
#### Post date: [February 7, 2020, 9:03pm UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120/2 "2020-02-07T21:03:17Z")
All DL4J CNN 2D layers - including those obtained from Keras import - expect NCHW format data. ImageRecordReader/NativeImageLoader produces NCHW data by default. So this should not be the issue. Can you upload the model?
There are MultiDataSetIterator that you can build and use for (3).
---
### Author:  [@j1234d2003](https://community.konduit.ai/u/j1234d2003)
#### Post date: [February 7, 2020, 10:15pm UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120/3 "2020-02-07T22:15:50Z")
Here is the model generation and training
```python
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.summary()
model.compile(loss='binary_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-4),
metrics=['acc'])
trainDataGenerator = ImageDataGenerator( rescale=1./255, rotation_range=40, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
trainDir = baseDir + "/train"
trainMetaFile = trainDir + "/train.csv"
trainDf = pd.read_csv(trainMetaFile, dtype=str)
trainGenerator=trainDataGenerator.flow_from_dataframe( dataframe=trainDf, directory=trainDir, x_col="file", y_col="label", batch_size=20, seed=42, shuffle=True, classes=CLASSES, class_mode="binary", target_size=(150, 150))
validationDir = baseDir + "/validation"
validationMetaFile = validationDir + "/validation.csv"
validationDf = pd.read_csv(validationMetaFile, dtype=str)
validationGenerator=trainDataGenerator.flow_from_dataframe( dataframe=validationDf, directory=validationDir, x_col="file", y_col="label", batch_size=20, seed=42, shuffle=True, classes=["dog", "cat"], class_mode="binary", target_size=(150, 150))
model.fit_generator( trainGenerator, steps_per_epoch=100, epochs=100,
validation_data=validationGenerator, validation_steps=50)
```
edit: make code snippets readable
---
### Author:  [@j1234d2003](https://community.konduit.ai/u/j1234d2003)
#### Post date: [February 8, 2020, 6:51am UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120/4 "2020-02-08T06:51:38Z")
I retrained the above model and retried - still same problem. Please find the trained model H5, test images and test programs below:
A] Model and data
Download the model from [cnnSample1Model.h5 - Google Drive](https://drive.google.com/open?id=1m57YG1_FfNxIQq1v6zSOScwy9hlVBWzs) and save it as /tmp/cnnSample1Model.h5
Download and untar the test data images from [testData.tar.gz - Google Drive](https://drive.google.com/open?id=1-9LTXFGyr5SMn4PkSpFZhdKaKorI8a7E) under /tmp/testData
B] DL4J Test Program:
* * *
```java
package com.sample;
import static edu.sciproj.libcore.CCommandLineOption.nonEmpty;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.nn.modelimport.keras.KerasModelImport;
import org.deeplearning4j.nn.modelimport.keras.exceptions.InvalidKerasConfigurationException;
import org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
import org.nd4j.linalg.dataset.api.preprocessor.ImagePreProcessingScaler;
import org.nd4j.linalg.factory.Nd4j;
// MultiDataSet handles multiple inputs and outputs.
public class TestCnn
{
private static final int[] TARGET_SIZE = new int[]{150, 150}; // w, h
private static final int IMAGE_CHANNEL = 3;
private String mInputPath;
public TestCnn(String modelFile, String inputDir)
{
mInputPath = inputDir;
}
public INDArray getInput() throws IOException
{
String[] testImgs = new String[]{"cat.1500.jpg","cat.1501.jpg","dog.1500.jpg","dog.1501.jpg"};
INDArray[] ret = new INDArray[testImgs.length];
int count = 0;
for (String testImg: testImgs)
{
File f = new File(mInputPath, testImg);
NativeImageLoader loader = new NativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL);
INDArray image = loader.asMatrix(f);
DataNormalization scalar = new ImagePreProcessingScaler(0, 1);
scalar.transform(image);
ret[count] = image;
count++;
}
// Concatenate the array along dimension 0.
return(Nd4j.concat(0, ret));
}
public MultiLayerNetwork loadModel(String modelFile)
throws IOException,
InvalidKerasConfigurationException,
UnsupportedKerasConfigurationException
{
return(KerasModelImport.importKerasSequentialModelAndWeights(modelFile));
}
public static void main(String[] args)
throws Exception
{
String homeDir = System.getenv("HOME");
String modelFile = "/tmp/cnnSample1Model.h5";
String inputDir = "/tmp/testData";
TestCnn loader = new TestCnn(modelFile, inputDir);
loader.getInput();
MultiLayerNetwork model = loader.loadModel(modelFile);
INDArray features = loader.getInput();
long[] shape = features.shape();
System.out.println("Feature shape=" + Arrays.toString(shape));
INDArray output = model.output(features);
System.out.println("Prediction: " + output);
}
}
```
* * *
Output:
Prediction: [[0.0426],
[0.8121],
[0.7653],
[0.8947]]
* * *
C] Keras Test Program
```python
import pandas as pd
import numpy as np
import sys
import traceback
from keras.models import load_model
from keras.preprocessing.image import ImageDataGenerator
def main(argv):
try:
model = load_model("/tmp/cnnSample1Model.h5")
imgGenerator = ImageDataGenerator(rescale=1./255)
testDir = "/tmp/testData"
testData = np.array([["cat.1500.jpg"],["cat.1501.jpg"],["dog.1500.jpg"],["dog.1501.jpg"]])
testDf = pd.DataFrame(testData, columns=["file"])
itr=imgGenerator.flow_from_dataframe(dataframe=testDf, directory=testDir,
x_col="file", y_col=None, batch_size=testData.shape[0], seed=42,
shuffle=False, class_mode=None, target_size=(150, 150))
netOutput = model.predict(itr);
print("Prediction: ", netOutput)
except Exception as excp:
print("Failed processing")
print(excp)
traceback.print_exc()
sys.exit(1)
if __name__ == ' __main__':
main(sys.argv)
```
* * *
Prediction: [[0.01988904]
[0.8130257]
[0.7170239]
[0.9342622]]
As you can see, the DL4J and Keras outputs are not identical for same test
images. Please help.
edit: make code snippets readable
---
### Author:  [@treo](https://community.konduit.ai/u/treo)
#### Post date: [February 26, 2020, 1:18pm UTC](https://community.konduit.ai/t/imported-pretrained-keras-model-produces-wrong-output-in-dl4j/120/5 "2020-02-26T13:18:05Z")
Everything points to a difference between RBG and BGR.
`NaviveImageLoader` loads images in the format that OpenCV loads them: BGR.
Unfortunately, there is no direct option to tell it to reverse this. However, `Java2DNativeImageLoader` does support it.
By modifiying your code to use it and request channel flipping like this, we get a result that is closer to what you have shown.
```java
Java2DNativeImageLoader javaLoader = new Java2DNativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL);
INDArray image = javaLoader.asMatrix(ImageIO.read(f), true);
```
```auto
Prediction: [[0.0194],
[0.8040],
[0.7122],
[0.9308]]
```
Edit:
As I’ve just learned, there is actually a simple way to get the RGB images from `NativeImageLoader`:
```auto
NativeImageLoader javaLoader = new NativeImageLoader(TARGET_SIZE[1], TARGET_SIZE[0], IMAGE_CHANNEL, new ColorConversionTransform(COLOR_BGR2RGB));
```
By passing the `ColorConversionTransform` like that, it will transform the channels into the order that your network expects them.
There are many other Transforms, that would also support transformations to other schemes like HSV, HSL or Luv.