Unsupported keras layer type UpSampling3D

FanDev · May 29, 2020, 2:39pm

Issue Description
I saw DL4J supports Upsampling3D but when I tried to load my h5 model, an error popped up as:

Exception in thread “main” org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException: Unsupported keras layer type UpSampling3D. Please file an issue at Issues · deeplearning4j/deeplearning4j · GitHub.
at org.deeplearning4j.nn.modelimport.keras.utils.KerasLayerUtils.getKerasLayerFromConfig(KerasLayerUtils.java:334)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.prepareLayers(KerasModel.java:218)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.(KerasModel.java:164)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.(KerasModel.java:96)
at org.deeplearning4j.nn.modelimport.keras.utils.KerasModelBuilder.buildModel(KerasModelBuilder.java:307)
at org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasModelAndWeights(KerasModelImport.java:172)
at org.deeplearning4j.examples.convolution.captcharecognition.FaultRecognition.main(FaultRecognition.java:90)

Version Information

Deeplearning4j version
1.0.0-beta6
Platform information (OS, etc) AWS Linux
No CUDA

Additional Information
My simple u-net model was build in original keras not tf.keras.

from keras.models import *
from keras.layers import *
from keras.optimizers import *
from keras.callbacks import ModelCheckpoint, LearningRateScheduler
from keras import backend as keras
from keras import metrics

treo · May 29, 2020, 3:15pm

Try again with beta7.

FanDev · June 6, 2020, 6:02pm

Sorry for reply so late, I am running beta-7 version, but still have same issue.

I am loading model as:

String fullModel = “Model.hdf5”;
ComputationGraph model = KerasModelImport.importKerasModelAndWeights(fullModel);

<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-examples-parent</artifactId>
<version>1.0.0-beta7</version>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>

<name>DeepLearning4j Examples Parent</name>
<description>Examples of training different data sets</description>
<properties>
    <!-- Change the nd4j.backend property to nd4j-cuda-10.0-platform, nd4j-cuda-10.1-platform or nd4j-cuda-10.2-platform to use CUDA GPUs -->
    <nd4j.backend>nd4j-native-platform</nd4j.backend>
    <!--<nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend>-->
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <shadedClassifier>bin</shadedClassifier>


    <java.version>1.8</java.version>
    <nd4j.version>1.0.0-beta7</nd4j.version>
    <dl4j.version>1.0.0-beta7</dl4j.version>
    <datavec.version>1.0.0-beta7</datavec.version>
    <arbiter.version>1.0.0-beta7</arbiter.version>
    <rl4j.version>1.0.0-beta7</rl4j.version>

    <!-- Scala binary version: DL4J's Spark and UI functionality are released with both Scala 2.10 and 2.11 support -->
    <scala.binary.version>2.11</scala.binary.version>
    <spark.version>2.4.3</spark.version>

    <hadoop.version>2.2.0</hadoop.version>  <!-- Hadoop version used by Spark 1.6.3 and 2.2.1 (and likely others) -->
    <guava.version>19.0</guava.version>
    <logback.version>1.1.7</logback.version>
    <jfreechart.version>1.0.13</jfreechart.version>
    <jcommon.version>1.0.23</jcommon.version>
    <maven-compiler-plugin.version>3.6.1</maven-compiler-plugin.version>
    <maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
    <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
    <maven.minimum.version>3.3.1</maven.minimum.version>
    <javafx.version>2.2.3</javafx.version>
    <javafx.runtime.lib.jar>${env.JAVAFX_HOME}/jfxrt.jar</javafx.runtime.lib.jar>
    <aws.sdk.version>1.11.109</aws.sdk.version>
    <jackson.version>2.5.1</jackson.version>
    <scala.plugin.version>3.2.2</scala.plugin.version>
</properties>

treo · June 8, 2020, 8:07am

That is weird, we should be supporting that layer in beta7 as far as I can tell.
Can you share your model file, or a script to create an equivalent model, so we can debug the issue?

FanDev · June 8, 2020, 3:07pm

Sure, here is how I implement:

Construct a simple Unet model by:

def unet(pretrained_weights = None,input_size = (128,128,128,1),DropoutRatio=0.5 ):
inputs = Input(input_size)

conv1 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(inputs) # kernel_initializer = 'he_normal'
conv1 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(conv1) 
pool1 = MaxPooling3D(pool_size=(2, 2, 2))(conv1)

conv2 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(pool1) 
conv2 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(conv2) 
pool2 = MaxPooling3D(pool_size=(2, 2, 2))(conv2)

conv3 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(pool2)
conv3 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(conv3)
pool3 = MaxPooling3D(pool_size=(2, 2, 2))(conv3)

conv4 = Conv3D(128, (3,3,3), activation = 'relu', padding = 'same')(pool3)
conv4 = Conv3D(128, (3,3,3), activation = 'relu', padding = 'same')(conv4)

merge5 = concatenate([UpSampling3D(size=(2,2,2))(conv4),conv3])
conv5 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(merge5)
conv5 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(conv5)

merge6 = concatenate([UpSampling3D(size=(2,2,2))(conv5),conv2])
conv6 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(merge6)
conv6 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(conv6)      

merge7 = concatenate([UpSampling3D(size=(2,2,2))(conv6),conv1])
conv7 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(merge7)
conv7 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(conv7)

conv8 = Conv3D(1, (1,1,1), activation = 'sigmoid')(conv7)

model = Model(input = inputs, output = conv8)
model.compile(optimizer = Adam(lr = 1e-4), loss = modified_crossentropy, metrics = ['accuracy'])   
model.summary()
return model

build a model as:

import modelBuilder
model = modelBuilder.unet(input_size = input_size, DropoutRatio= DropoutRatio)

I saved my model by running:

model_checkpoint = ModelCheckpoint(‘model_saved_to_disk.hdf5’, monitor=‘loss’,verbose=1, save_best_only=True)

Going into deeplearning4j, My project is original "

deeplearning4j-examples

use

mvn install clean

to build this project successfully.

Then I used MultiDigitNumberRecognition.java as a template to create a new java in the folder of dl4j-examples/src/main/java/org/deeplearning4j/examples/convolution/captcharecognition, called test.java.

I added following code and tried to load my model first:

String downloadPath = "model_saved_to_disk.hdf5";
 File cachedKerasFile = new File(downloadPath);
ComputationGraph model1 = KerasModelImport.importKerasModelAndWeights(cachedKerasFile.getAbsolutePath());

I am new to DL4J, I may miss some configuration or incorrect setup? Appreciated if you can point out any issue I have.

FanDev · June 13, 2020, 2:47pm

Any solutions? I tried to read other model from someone else but will upsampling3D, still didn’t work.

treo · June 13, 2020, 3:03pm

Unfortunately I didn’t have time yet to look into it

FanDev · June 14, 2020, 9:15pm

HI, I think I may find out what’s going on there, may be a bug for this class KerasLayerUtils.java forgot to include getLAYER_CLASS_NAME_ZERO_UPSAMPLING_3D().

Is there any thing I can do to modify this? or to add getLAYER_CLASS_NAME_ZERO_UPSAMPLING_3D to Java code?

Thanks.

treo · June 15, 2020, 6:03am

Thank you very much for delving into the code yourself and finding the cause of the bug. I’ve created an issue to track the fix of the bug:

github.com/eclipse/deeplearning4j

KerasUpsampling3D Layer isn't correctly imported

opened 06:02AM - 15 Jun 20 UTC

closed 05:01AM - 27 Jan 21 UTC

treo

Bug DL4J Keras

#### Issue Description Originally reported at https://community.konduit.ai/t/un…supported-keras-layer-type-upsampling3d/564/7 We actually implement the layer here: https://github.com/eclipse/deeplearning4j/blob/b06fb670a420050f1edcdc398f11fe76fadf1538/deeplearning4j/deeplearning4j-modelimport/src/main/java/org/deeplearning4j/nn/modelimport/keras/layers/convolutional/KerasUpsampling3D.java But during import we don't check for it: https://github.com/eclipse/deeplearning4j/blob/b06fb670a420050f1edcdc398f11fe76fadf1538/deeplearning4j/deeplearning4j-modelimport/src/main/java/org/deeplearning4j/nn/modelimport/keras/utils/KerasLayerUtils.java#L296-L299 As a workaround the following **should** work (haven't tested it yet though): ``` KerasLayer.registerCustomLayer("UpSampling3D", KerasUpsampling3D.class); ``` #### Version Information Please indicate relevant versions, including, if relevant: * Deeplearning4j version: 1.0.0-beta7 * Platform information (OS, etc): all * CUDA version, if used: N/A * NVIDIA driver version, if in use: N/A

As a workaround you should be able to register the layer as a custom layer (see https://deeplearning4j.konduit.ai/keras-import/custom-layers#keraslayer)

I haven’t tried the following code, but I think it should work:

KerasLayer.registerCustomLayer("UpSampling3D", KerasUpsampling3D.class);

FanDev · June 15, 2020, 1:22pm

Thank you so much. It works! Still trying to understand more features from DL4J, will ask more questions then:)

Topic		Replies	Views
Unsupported keras layer type MyCrossLayer DL4J	4	422	August 11, 2020
Importing a Keras model into DL4J DL4J	5	375	June 8, 2023
Errors when Trying to import Keras Model into Java DL4J	12	355	August 13, 2024
Can not import models from keras DL4J	12	1021	May 13, 2020
Error importing tf.keras Xception Model DL4J	5	709	March 19, 2020

Unsupported keras layer type UpSampling3D

Related topics