most citedLatent Multi-Head Attention for Small Language Models

1 citations · 1 across the 3 of their papers we have counts for

collaborators
Showing cs.AIShow all

1 paper · 1 filter

cs.AI2025

Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models

Sushant Mehta, Raj Dandekar, Rajat Dandekar +1

We present MoE-MLA-RoPE, a novel architecture combination that combines Mixture of Experts (MoE) with Multi-head Latent Attention (MLA) and Rotary Position Embeddings (RoPE) for ef…