Scaling Embedding Layers in Language Models

Da Yu, Edith Cohen, Badih Ghazi +5

We propose $SCONE$ ($S$calable, $C$ontextualized, $O$ffloaded, $N$-gram $E$mbedding), a new method for extending input embedding layers to enhance language model performance. To av…