Rewiring the Transformer with Depth-Wise LSTMs

Hongfei Xu, Yang Song, Qiuhui Liu +2

Stacking non-linear layers allows deep neural networks to model complicated functions, and including residual connections in Transformer layers is beneficial for convergence and pe…