Latent Multi-Head Attention for Small Language Models

Sushant Mehta, Raj Dandekar, Rajat Dandekar +1

We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs. Training 30M-parameter…