Sparse Universal Transformer

Shawn Tan, Yikang Shen, Zhenfang Chen +2

The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers. Empirical evidence shows that UTs have better compositional generalization…