Transformer's Encoder with Self-Attention layers

[treo]

I’ve answered the question on when and how to apply a mask in SameDiff over at your other thread: Feature Mask application in custom SameDiff model - #2 by treo

@partarstu

But BP is my problem. I haven’t found any available solution to force the BP in SameDiff into using the feature masks. Do you have any suggestions how I could workaround this issue?

It isn’t necessary for you to do that. They will be automatically applied during backprop with SameDiff when you apply it in the forward pass. The entire point of SameDiff is that it does the entire backprop for you so you don’t have to define it manually.