Layer Normalization

DL4J
1

Hi,

i wanted to ask i there are currently any development in this direction.

Especially for recurrent and attention layer it would be a nice to have option.

Best regards

Thomas

2

Ok, i tried base implementation as an SameDiffVertex without learnable parameter with help of:

a short test with SameDiff:

@Override
public SDVariable defineVertex(SameDiff sameDiff, VertexInputs inputs) {
	
    SDVariable x1 = inputs.getInput(0);

    // mean_i = sum(x_i[j] for j in range(k)) / k
    SDVariable mean1 = x1.mean(0);
    // var_i = sum((x_i[j] - mean_i) ** 2 for j in range(k)) / k
    SDVariable var1 = x1.sub(mean1).pow(2).mean(0);
    // x_i_normalized = (x_i - mean_i) / sqrt(var_i + epsilon)
    SDVariable norm1 = x1.sub(mean1).div(sameDiff.math.square(var1.add(1e-10)));

    return norm1;
}

Can anybody give me a short hint how this would map to an RNN Input Matrix. I know i get the base variable inside the defineVertex Method with something like that:

inputs.getInput(0)

or is this already enough if i use: x1 = inputs.getInput(0);

Appreciate any hint.

Best regards

thomas

3

Layer Normalization is already implemented in SameDiff:

You can easily use it with sd.nn.layerNorm (see also https://deeplearning4j.konduit.ai/samediff/reference/operation-namespaces/nn#layernorm)

4

Thanks, i will try to integrate into my layer.