General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings

Lukas Galke, Isabelle Cuber, Christoph Meyer +3

Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks. However, their sheer size is prohibitive for small laboratories or…