From the 1 of 4 linked papers with an AI index.
4 papers
Raven: High-Recall Sequence Modeling with Sparse Memory Routing
Arshia Afzal, Aviv Bick, Eric P. Xing +2
Raven is a linear-time sequence model that uses learned, input-dependent routing to update only a subset of fixed memory slots, reducing interference and improving long-range recal…
Retrieval-Aware Distillation for Transformer-SSM Hybrids
Aviv Bick, Eric P. Xing, Albert Gu
State-space models (SSMs) offer efficient sequence modeling but lag behind Transformers on benchmarks that require in-context retrieval. Prior work links this gap to a small set of…
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
Aviv Bick, Eric Xing, Albert Gu
State-space models (SSMs) offer efficient alternatives to Transformers for long sequences, but their fixed-size recurrent state limits capability on algorithmic tasks, such as retr…
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
Aviv Bick, Kevin Y. Li, Eric P. Xing +2
Transformer architectures have become a dominant paradigm for domains like language modeling but suffer in many inference settings due to their quadratic-time self-attention. Recen…