157 citations · 476 across the 27 of their papers we have counts for
5 papers · 1 filter
Sliding Window Recurrences for Sequence Models
Dragos Secrieru, Garyk Brixi, Yoshua Bengio +3
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrenc…
VRPAgent: LLM-Driven Discovery of Heuristic Operators for Vehicle Routing Problems
André Hottung, Federico Berto, Chuanbo Hua +9
Designing high-performing heuristics for vehicle routing problems (VRPs) is a complex task that requires both intuition and deep domain knowledge. Large language model (LLM)-based…
Exploring Diffusion Transformer Designs via Grafting
Keshigeyan Chandrasegaran, Michael Poli, Daniel Y. Fu +9
Designing model architectures requires decisions such as selecting operators (e.g., attention, convolution) and configurations (e.g., depth, width). However, evaluating the impact…
Quantifying Memory Utilization with Effective State-Size
Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas +6
The need to develop a general framework for architecture analysis is becoming increasingly important, given the expanding design space of sequence models. To this end, we draw insi…
Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
Jerome Ku, Eric Nguyen, David W. Romero +13
We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation ta…