MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings

Andrea Gurioli, Federico Pennino, João Monteiro +1

Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces…