14 citations · 14 across the 3 of their papers we have counts for
4 papers
Quantifying Memory Utilization with Effective State-Size
Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas +6
The need to develop a general framework for architecture analysis is becoming increasingly important, given the expanding design space of sequence models. To this end, we draw insi…
STAR: Synthesis of Tailored Architectures
Armin W. Thomas, Rom Parnichkun, Alexander Amini +2
Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality…
Mechanistic Design and Scaling of Hybrid Architectures
Michael Poli, Armin W Thomas, Eric Nguyen +9
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale…
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
Daniel Y. Fu, Simran Arora, Jessica Grogan +7
Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures s…