463 citations · 1.5k across the 29 of their papers we have counts for
29 papers
Dion3: Full-Stack Orthogonal Updates
Noah Amsel, Jack Zhang, Kwangjun Ahn +5
The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this co…
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Zichang Liu, Jue Wang, Tri Dao +8
Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference t…
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Tri Dao
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image…
StarCoder: may the source be with you!
Raymond Li, Loubna Ben Allal, Yangtian Zi +64
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase…
Effectively Modeling Time Series with Simple Discrete State Spaces
Michael Zhang, Khaled K. Saab, Michael Poli +3
Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) effic…
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen +5
State space models (SSMs) have high performance on long sequence modeling but require sophisticated initialization techniques and specialized implementations for high quality and r…