MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

Muchen Li, Leonid Sigal, Renjie Liao

Scaling large language models efficiently has motivated sparse capacity mechanisms such as Mixture-of-Experts and, more recently, conditional memory: token-indexed embedding tables…