Squeeze-and-Excitation Network is an interesting publication:
There are many implementations online, e.g. in TensorFlow see se_bottleneck:
It looks like it should be possible to implement it in DL4J, but I’m not sure which layers to use.
I guess first would be a GlobalPoolingLayer with “collapseDimensions = false”.
Then CnnToFeedForwardPreProcessor, and 2 x DenseLayer.
Then we need to scale and add, see “SE-ResNet Module” picture in publications.
For that, do we use a FeedForwardToCnnPreProcessor into Upsampling2D and a ScaleVertex?
I haven’t used the Upsampling2D or ScaleVertex before, do they make sense here or do we need something else?
It would be great to have an example of this on deeplearning4j-examples.