Squeeze-and-Excitation Network example (SENet / SE-ResNet)

alexoooo · February 19, 2020, 10:43pm

Squeeze-and-Excitation Network is an interesting publication:

There are many implementations online, e.g. in TensorFlow see se_bottleneck:

tensorpack/tensorpack/blob/master/examples/ResNet/resnet_model.py

# -*- coding: utf-8 -*-
# File: resnet_model.py

import tensorflow as tf
from tensorpack import tfv1
from tensorpack.models import BatchNorm, BNReLU, Conv2D, FullyConnected, GlobalAvgPooling, MaxPooling
from tensorpack.tfutils.argscope import argscope, get_arg_scope


def resnet_shortcut(l, n_out, stride, activation=tf.identity):
    data_format = get_arg_scope()['Conv2D']['data_format']
    n_in = l.get_shape().as_list()[1 if data_format in ['NCHW', 'channels_first'] else 3]
    if n_in != n_out:   # change dimension when channel is not the same
        return Conv2D('convshortcut', l, n_out, 1, strides=stride, activation=activation)
    else:
        return l


def get_bn(zero_init=False):
    """

This file has been truncated. show original

It looks like it should be possible to implement it in DL4J, but I’m not sure which layers to use.

I guess first would be a GlobalPoolingLayer with “collapseDimensions = false”.
Then CnnToFeedForwardPreProcessor, and 2 x DenseLayer.

Then we need to scale and add, see “SE-ResNet Module” picture in publications.

For that, do we use a FeedForwardToCnnPreProcessor into Upsampling2D and a ScaleVertex?
I haven’t used the Upsampling2D or ScaleVertex before, do they make sense here or do we need something else?

It would be great to have an example of this on deeplearning4j-examples.

Topic		Replies	Views
Generic windowing functions for 2D images? DL4J	2	392	July 8, 2020
Temporal Convolutional Network DL4J	11	784	February 6, 2020
Stacking c copies of a (1,m,n) layer into a (c,m,n) tensor, inside network DL4J	5	184	July 15, 2023
Advice on non-square input DL4J	4	439	March 22, 2020
Simple CNN predicts NaNs DL4J	3	468	November 2, 2020

Squeeze-and-Excitation Network example (SENet / SE-ResNet)

Related Topics