157 citations · 476 across the 27 of their papers we have counts for
24 papers · 1 filter
Sliding Window Recurrences for Sequence Models
Dragos Secrieru, Garyk Brixi, Yoshua Bengio +3
Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrenc…
Exploring Diffusion Transformer Designs via Grafting
Keshigeyan Chandrasegaran, Michael Poli, Daniel Y. Fu +9
Designing model architectures requires decisions such as selecting operators (e.g., attention, convolution) and configurations (e.g., depth, width). However, evaluating the impact…
Quantifying Memory Utilization with Effective State-Size
Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas +6
The need to develop a general framework for architecture analysis is becoming increasingly important, given the expanding design space of sequence models. To this end, we draw insi…
Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
Jerome Ku, Eric Nguyen, David W. Romero +13
We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation ta…
STAR: Synthesis of Tailored Architectures
Armin W. Thomas, Rom Parnichkun, Alexander Amini +2
Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality…
State-Free Inference of State-Space Models: The Transfer Function Approach
Rom N. Parnichkun, Stefano Massaroli, Alessandro Moro +10
We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel in…
Mechanistic Design and Scaling of Hybrid Architectures
Michael Poli, Armin W Thomas, Eric Nguyen +9
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale…
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions
Stefano Massaroli, Michael Poli, Daniel Y. Fu +11
Recent advances in attention-free sequence models rely on convolutions as alternatives to the attention operator at the core of Transformers. In particular, long convolution sequen…
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Eric Nguyen, Michael Poli, Marjan Faizi +10
Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation…
Improving day-ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context
Oussama Boussif, Ghait Boukachab, Dan Assouline +4
Solar power harbors immense potential in mitigating climate change by substantially reducing CO emissions. Nonetheless, the inherent variability of solar irradiance poses a s…
Ideal Abstractions for Decision-Focused Learning
Michael Poli, Stefano Massaroli, Stefano Ermon +2
We present a methodology for formulating simplifying abstractions in machine learning systems by identifying and harnessing the utility structure of decisions. Machine learning tas…
Hyena Hierarchy: Towards Larger Convolutional Language Models
Michael Poli, Stefano Massaroli, Eric Nguyen +6
Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the…
Transform Once: Efficient Operator Learning in Frequency Domain
Michael Poli, Stefano Massaroli, Federico Berto +4
Spectral analysis provides one of the most effective paradigms for information-preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often…
Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations
Michael Poli, Winnie Xu, Stefano Massaroli +3
Many patterns in nature exhibit self-similarity: they can be compactly described via self-referential transformations. Said patterns commonly appear in natural and artificial objec…
Continuous-Depth Neural Models for Dynamic Graph Prediction
Michael Poli, Stefano Massaroli, Clayton M. Rabideau +4
We introduce the framework of continuous-depth graph neural networks (GNNs). Neural graph differential equations (Neural GDEs) are formalized as the counterpart to GNNs where the i…
Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions
Michael Poli, Stefano Massaroli, Luca Scimeca +6
Effective control and prediction of dynamical systems often require appropriate handling of continuous-time and discrete, event-triggered processes. Stochastic hybrid systems (SHSs…
Differentiable Multiple Shooting Layers
Stefano Massaroli, Michael Poli, Sho Sonoda +4
We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value prob…
Learning Stochastic Optimal Policies via Gradient Descent
Stefano Massaroli, Michael Poli, Stefano Peluchetti +3
We systematically develop a learning-based treatment of stochastic optimal control (SOC), relying on direct optimization of parametric control policies. We propose a derivation of…
Neural Ordinary Differential Equations for Intervention Modeling
Daehoon Gwak, Gyuhyeon Sim, Michael Poli +3
By interpreting the forward dynamics of the latent representation of neural networks as an ordinary differential equation, Neural Ordinary Differential Equation (Neural ODE) emerge…
TorchDyn: A Neural Differential Equations Library
Michael Poli, Stefano Massaroli, Atsushi Yamashita +2
Continuous-depth learning has recently emerged as a novel perspective on deep learning, improving performance in tasks related to dynamical systems and density estimation. Core to…
Hypersolvers: Toward Fast Continuous-Depth Models
Michael Poli, Stefano Massaroli, Atsushi Yamashita +2
The infinite-depth paradigm pioneered by Neural ODEs has launched a renaissance in the search for novel dynamical system-inspired deep learning primitives; however, their utilizati…
Stable Neural Flows
Stefano Massaroli, Michael Poli, Michelangelo Bin +3
We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network. S…
Dissecting Neural ODEs
Stefano Massaroli, Michael Poli, Jinkyoo Park +2
Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap…
Graph Neural Ordinary Differential Equations
Michael Poli, Stefano Massaroli, Junyoung Park +3
We introduce the framework of continuous--depth graph neural networks (GNNs). Graph neural ordinary differential equations (GDEs) are formalized as the counterpart to GNNs where th…