EELBERT: Tiny Models through Dynamic Embeddings

Gabrielle Cohn, Rishika Agarwal, Deepanshu Gupta +1

We introduce EELBERT, an approach for compression of transformer-based models (e.g., BERT), with minimal impact on the accuracy of downstream tasks. This is achieved by replacing t…