Layer Normalization


i wanted to ask i there are currently any development in this direction.

Especially for recurrent and attention layer it would be a nice to have option.

Best regards


Ok, i tried base implementation as an SameDiffVertex without learnable parameter with help of:

a short test with SameDiff:

public SDVariable defineVertex(SameDiff sameDiff, VertexInputs inputs) {
    SDVariable x1 = inputs.getInput(0);

    // mean_i = sum(x_i[j] for j in range(k)) / k
    SDVariable mean1 = x1.mean(0);
    // var_i = sum((x_i[j] - mean_i) ** 2 for j in range(k)) / k
    SDVariable var1 = x1.sub(mean1).pow(2).mean(0);
    // x_i_normalized = (x_i - mean_i) / sqrt(var_i + epsilon)
    SDVariable norm1 = x1.sub(mean1).div(sameDiff.math.square(var1.add(1e-10)));

    return norm1;

Can anybody give me a short hint how this would map to an RNN Input Matrix. I know i get the base variable inside the defineVertex Method with something like that:


or is this already enough if i use: x1 = inputs.getInput(0);

Appreciate any hint.

Best regards


Layer Normalization is already implemented in SameDiff:

You can easily use it with sd.nn.layerNorm (see also

Thanks, i will try to integrate into my layer.