L${}^3$: Large Lookup Layers

Albert Tseng, Christopher De Sa

Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routi…