Unsupported keras layer type UpSampling3D

FanDev · May 29, 2020, 2:39pm

Issue Description
I saw DL4J supports Upsampling3D but when I tried to load my h5 model, an error popped up as:

Exception in thread “main” org.deeplearning4j.nn.modelimport.keras.exceptions.UnsupportedKerasConfigurationException: Unsupported keras layer type UpSampling3D. Please file an issue at Issues · deeplearning4j/deeplearning4j · GitHub.
at org.deeplearning4j.nn.modelimport.keras.utils.KerasLayerUtils.getKerasLayerFromConfig(KerasLayerUtils.java:334)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.prepareLayers(KerasModel.java:218)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.(KerasModel.java:164)
at org.deeplearning4j.nn.modelimport.keras.KerasModel.(KerasModel.java:96)
at org.deeplearning4j.nn.modelimport.keras.utils.KerasModelBuilder.buildModel(KerasModelBuilder.java:307)
at org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasModelAndWeights(KerasModelImport.java:172)
at org.deeplearning4j.examples.convolution.captcharecognition.FaultRecognition.main(FaultRecognition.java:90)

Version Information

Deeplearning4j version
1.0.0-beta6
Platform information (OS, etc) AWS Linux
No CUDA

Additional Information
My simple u-net model was build in original keras not tf.keras.

from keras.models import *
from keras.layers import *
from keras.optimizers import *
from keras.callbacks import ModelCheckpoint, LearningRateScheduler
from keras import backend as keras
from keras import metrics

treo · May 29, 2020, 3:15pm

Try again with beta7.

FanDev · June 6, 2020, 6:02pm

Sorry for reply so late, I am running beta-7 version, but still have same issue.

I am loading model as:

String fullModel = “Model.hdf5”;
ComputationGraph model = KerasModelImport.importKerasModelAndWeights(fullModel);

<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-examples-parent</artifactId>
<version>1.0.0-beta7</version>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>

<name>DeepLearning4j Examples Parent</name>
<description>Examples of training different data sets</description>
<properties>
    <!-- Change the nd4j.backend property to nd4j-cuda-10.0-platform, nd4j-cuda-10.1-platform or nd4j-cuda-10.2-platform to use CUDA GPUs -->
    <nd4j.backend>nd4j-native-platform</nd4j.backend>
    <!--<nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend>-->
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <shadedClassifier>bin</shadedClassifier>


    <java.version>1.8</java.version>
    <nd4j.version>1.0.0-beta7</nd4j.version>
    <dl4j.version>1.0.0-beta7</dl4j.version>
    <datavec.version>1.0.0-beta7</datavec.version>
    <arbiter.version>1.0.0-beta7</arbiter.version>
    <rl4j.version>1.0.0-beta7</rl4j.version>

    <!-- Scala binary version: DL4J's Spark and UI functionality are released with both Scala 2.10 and 2.11 support -->
    <scala.binary.version>2.11</scala.binary.version>
    <spark.version>2.4.3</spark.version>

    <hadoop.version>2.2.0</hadoop.version>  <!-- Hadoop version used by Spark 1.6.3 and 2.2.1 (and likely others) -->
    <guava.version>19.0</guava.version>
    <logback.version>1.1.7</logback.version>
    <jfreechart.version>1.0.13</jfreechart.version>
    <jcommon.version>1.0.23</jcommon.version>
    <maven-compiler-plugin.version>3.6.1</maven-compiler-plugin.version>
    <maven-shade-plugin.version>2.4.3</maven-shade-plugin.version>
    <exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
    <maven.minimum.version>3.3.1</maven.minimum.version>
    <javafx.version>2.2.3</javafx.version>
    <javafx.runtime.lib.jar>${env.JAVAFX_HOME}/jfxrt.jar</javafx.runtime.lib.jar>
    <aws.sdk.version>1.11.109</aws.sdk.version>
    <jackson.version>2.5.1</jackson.version>
    <scala.plugin.version>3.2.2</scala.plugin.version>
</properties>

treo · June 8, 2020, 8:07am

That is weird, we should be supporting that layer in beta7 as far as I can tell.
Can you share your model file, or a script to create an equivalent model, so we can debug the issue?

FanDev · June 8, 2020, 3:07pm

Sure, here is how I implement:

Construct a simple Unet model by:

def unet(pretrained_weights = None,input_size = (128,128,128,1),DropoutRatio=0.5 ):
inputs = Input(input_size)

conv1 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(inputs) # kernel_initializer = 'he_normal'
conv1 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(conv1) 
pool1 = MaxPooling3D(pool_size=(2, 2, 2))(conv1)

conv2 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(pool1) 
conv2 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(conv2) 
pool2 = MaxPooling3D(pool_size=(2, 2, 2))(conv2)

conv3 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(pool2)
conv3 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(conv3)
pool3 = MaxPooling3D(pool_size=(2, 2, 2))(conv3)

conv4 = Conv3D(128, (3,3,3), activation = 'relu', padding = 'same')(pool3)
conv4 = Conv3D(128, (3,3,3), activation = 'relu', padding = 'same')(conv4)

merge5 = concatenate([UpSampling3D(size=(2,2,2))(conv4),conv3])
conv5 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(merge5)
conv5 = Conv3D(64, (3,3,3), activation = 'relu', padding = 'same')(conv5)

merge6 = concatenate([UpSampling3D(size=(2,2,2))(conv5),conv2])
conv6 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(merge6)
conv6 = Conv3D(32, (3,3,3), activation = 'relu', padding = 'same')(conv6)      

merge7 = concatenate([UpSampling3D(size=(2,2,2))(conv6),conv1])
conv7 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(merge7)
conv7 = Conv3D(16, (3,3,3), activation = 'relu', padding = 'same')(conv7)

conv8 = Conv3D(1, (1,1,1), activation = 'sigmoid')(conv7)

model = Model(input = inputs, output = conv8)
model.compile(optimizer = Adam(lr = 1e-4), loss = modified_crossentropy, metrics = ['accuracy'])   
model.summary()
return model

build a model as:

import modelBuilder
model = modelBuilder.unet(input_size = input_size, DropoutRatio= DropoutRatio)

I saved my model by running:

model_checkpoint = ModelCheckpoint(‘model_saved_to_disk.hdf5’, monitor=‘loss’,verbose=1, save_best_only=True)

Going into deeplearning4j, My project is original "

deeplearning4j-examples

use

mvn install clean

to build this project successfully.

Then I used MultiDigitNumberRecognition.java as a template to create a new java in the folder of dl4j-examples/src/main/java/org/deeplearning4j/examples/convolution/captcharecognition, called test.java.

I added following code and tried to load my model first:

String downloadPath = "model_saved_to_disk.hdf5";
 File cachedKerasFile = new File(downloadPath);
ComputationGraph model1 = KerasModelImport.importKerasModelAndWeights(cachedKerasFile.getAbsolutePath());

I am new to DL4J, I may miss some configuration or incorrect setup? Appreciated if you can point out any issue I have.

FanDev · June 13, 2020, 2:47pm

Any solutions? I tried to read other model from someone else but will upsampling3D, still didn’t work.

treo · June 13, 2020, 3:03pm

Unfortunately I didn’t have time yet to look into it

FanDev · June 14, 2020, 9:15pm

HI, I think I may find out what’s going on there, may be a bug for this class KerasLayerUtils.java forgot to include getLAYER_CLASS_NAME_ZERO_UPSAMPLING_3D().

Is there any thing I can do to modify this? or to add getLAYER_CLASS_NAME_ZERO_UPSAMPLING_3D to Java code?

Thanks.

treo · June 15, 2020, 6:03am

Thank you very much for delving into the code yourself and finding the cause of the bug. I’ve created an issue to track the fix of the bug:

github.com/eclipse/deeplearning4j

KerasUpsampling3D Layer isn't correctly imported

opened 06:02AM - 15 Jun 20 UTC

closed 05:01AM - 27 Jan 21 UTC

treo

Bug DL4J Keras

#### Issue Description Originally reported at https://community.konduit.ai/t/un…supported-keras-layer-type-upsampling3d/564/7 We actually implement the layer here: https://github.com/eclipse/deeplearning4j/blob/b06fb670a420050f1edcdc398f11fe76fadf1538/deeplearning4j/deeplearning4j-modelimport/src/main/java/org/deeplearning4j/nn/modelimport/keras/layers/convolutional/KerasUpsampling3D.java But during import we don't check for it: https://github.com/eclipse/deeplearning4j/blob/b06fb670a420050f1edcdc398f11fe76fadf1538/deeplearning4j/deeplearning4j-modelimport/src/main/java/org/deeplearning4j/nn/modelimport/keras/utils/KerasLayerUtils.java#L296-L299 As a workaround the following **should** work (haven't tested it yet though): ``` KerasLayer.registerCustomLayer("UpSampling3D", KerasUpsampling3D.class); ``` #### Version Information Please indicate relevant versions, including, if relevant: * Deeplearning4j version: 1.0.0-beta7 * Platform information (OS, etc): all * CUDA version, if used: N/A * NVIDIA driver version, if in use: N/A

As a workaround you should be able to register the layer as a custom layer (see https://deeplearning4j.konduit.ai/keras-import/custom-layers#keraslayer)

I haven’t tried the following code, but I think it should work:

KerasLayer.registerCustomLayer("UpSampling3D", KerasUpsampling3D.class);

FanDev · June 15, 2020, 1:22pm

Thank you so much. It works! Still trying to understand more features from DL4J, will ask more questions then:)

Topic		Replies	Views
Unsupported keras layer type MyCrossLayer DL4J	4	398	August 11, 2020
Importing a Keras model into DL4J DL4J	5	235	June 8, 2023
Can not import models from keras DL4J	12	900	May 13, 2020
Errors when Trying to import Keras Model into Java DL4J	7	163	March 1, 2024
Error importing tf.keras Xception Model DL4J	5	661	March 19, 2020

Unsupported keras layer type UpSampling3D

Related Topics