1 citations · 1 across the 4 of their papers we have counts for
4 papers · 1 filter
Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
Zheng Zhan, Liliang Ren, Shuohang Wang +5
Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances,…
Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size
Soufiane Hayou, Liyuan Liu
Pretraining large language models is a costly process. To make this process more efficient, several methods have been proposed to optimize model architecture/parametrization and ha…
An Introduction to Discrete Variational Autoencoders
Alan Jeffares, Liyuan Liu
Variational Autoencoders (VAEs) are well-established as a principled approach to probabilistic unsupervised learning with neural networks. Typically, an encoder network defines the…
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
Rongzhi Zhang, Kuang Wang, Liyuan Liu +4
The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed K…