Parameter-Efficient Transformer Embeddings

Henry Ndubuaku, Mouad Talhi

Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains propor…