Rethinking Weight Tying: Pseudo-Inverse Tying for LM Stable Training and Updates

Jian Gu, Aldeida Aleti, Chunyang Chen +1

Weight tying is widely used in compact language models to reduce parameters by sharing the token table between the input embedding and the output projection. However, parameter sha…