Using the Output Embedding to Improve Language Models

Ofir Press, Lior Wolf

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying…