# Layer Normalization

**URL:** <https://community.konduit.ai/t/layer-normalization/2187>\
**Category:** DL4J\
**Created:** [January 5, 2023, 2:27pm UTC](https://community.konduit.ai/t/layer-normalization/2187 "2023-01-05T14:27:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![thomas](https://avatars.discourse-cdn.com/v4/letter/t/e36b37/32.png) [@thomas](https://community.konduit.ai/u/thomas)\
**Post date:** [January 5, 2023, 2:27pm UTC](https://community.konduit.ai/t/layer-normalization/2187/1 "2023-01-05T14:27:13Z")

</div>

Hi,

i wanted to ask i there are currently any development in this direction.

> <https://github.com/deeplearning4j/deeplearning4j/issues/4829>
>
> \#### Issue Description
> 
> Do Layer Normalization provided by DL4j?
> Or any repla…cement of it?
> 
> \#### Version Information
> 
> Please indicate relevant versions, including, if relevant:
> 
> \* Deeplearning4j version : 0.9.1

> <https://github.com/deeplearning4j/deeplearning4j/issues/5840>
>
> Recent work trying to understand why Batch Norm (BN) works shows (https://arxiv.…org/pdf/1805.11604.pdf) that there are other ways to achieve the same benefits using simpler methods.
> 
> Importantly in practice, BN can't be used with dropout (unstable). And this prevents getting confidence ranges using dropout on networks using BN.
> 
> I emailed one of the lead authors of the paper above to learn how to implement the l\_1 norm alternative to BN, and here is the strategy:
> 
> \`\`\`
> norm = compute\_l1\_norm(x\_1,...,x\_n)
> mean = compute\_mean(x\_1,...,x\_n)
> xHat\_i = (x\_i - mean) / norm
> \`\`\`
> 
> I presume this would be pretty straight forward to implement, because it's simpler than BN. I'm just not familiar with how you'd actually set this up in DL4J.
> 
> Aha! Link: https://skymindai.aha.io/features/DL4J-87

Especially for recurrent and attention layer it would be a nice to have option.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![thomas](https://avatars.discourse-cdn.com/v4/letter/t/e36b37/32.png) [@thomas](https://community.konduit.ai/u/thomas)\
**Post date:** [January 8, 2023, 8:59pm UTC](https://community.konduit.ai/t/layer-normalization/2187/2 "2023-01-08T20:59:38Z")

</div>

Ok, i tried base implementation as an SameDiffVertex without learnable parameter with help of:

> **[Keras documentation: LayerNormalization layer](https://keras.io/api/layers/normalization_layers/layer_normalization/)**
>
> Keras documentation

a short test with SameDiff:

```
@Override
public SDVariable defineVertex(SameDiff sameDiff, VertexInputs inputs) {
	
    SDVariable x1 = inputs.getInput(0);

    // mean_i = sum(x_i[j] for j in range(k)) / k
    SDVariable mean1 = x1.mean(0);
    // var_i = sum((x_i[j] - mean_i) ** 2 for j in range(k)) / k
    SDVariable var1 = x1.sub(mean1).pow(2).mean(0);
    // x_i_normalized = (x_i - mean_i) / sqrt(var_i + epsilon)
    SDVariable norm1 = x1.sub(mean1).div(sameDiff.math.square(var1.add(1e-10)));

    return norm1;
}

```

Can anybody give me a short hint how this would map to an RNN Input Matrix. I know i get the base variable inside the defineVertex Method with something like that:

inputs.getInput(0)

or is this already enough if i use: x1 = inputs.getInput(0);

Appreciate any hint.

Best regards

thomas

---

<div class="post-metadata">

**Author:** ![treo](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/treo/32/47_2.png) [@treo](https://community.konduit.ai/u/treo)\
**Post date:** [January 9, 2023, 7:49am UTC](https://community.konduit.ai/t/layer-normalization/2187/3 "2023-01-09T07:49:40Z")

</div>

Layer Normalization is already implemented in SameDiff:

> <https://github.com/deeplearning4j/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/api/ops/impl/transforms/custom/LayerNorm.java#L41>

You can easily use it with `sd.nn.layerNorm` (see also [https://deeplearning4j.konduit.ai/samediff/reference/operation-namespaces/nn#layernorm](https://deeplearning4j.konduit.ai/samediff/reference/operation-namespaces/nn#layernorm))

---

<div class="post-metadata">

**Author:** ![thomas](https://avatars.discourse-cdn.com/v4/letter/t/e36b37/32.png) [@thomas](https://community.konduit.ai/u/thomas)\
**Post date:** [January 9, 2023, 3:40pm UTC](https://community.konduit.ai/t/layer-normalization/2187/4 "2023-01-09T15:40:56Z")

</div>

Thanks, i will try to integrate into my layer.