Multi-Sense Embeddings for Language Models and Knowledge Distillation

Qitong Wang, Mohammed J. Zaki, Georgios Kollias +1

Transformer-based large language models (LLMs) rely on contextual embeddings which generate different (continuous) representations for the same token depending on its surrounding c…