Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models

Isaac Ellmen, David Errington, Matthew I. J. Raybould +1

Although $\mathbb{E}[\mathrm{dropout}(x)]=x$, here we show that $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))]$ is not equal to $\mathrm{LayerNorm}(x)$. Accordingly, the pa…