Leviathan: Decoupling Input and Output Representations in Language Models

Reza T. Batley, Sourav Saha

Modern language models use a single matrix for input embedding and output projection. This couples two distinct objectives: token representation and discrimination over a vocabular…