82 citations · 145 across the 26 of their papers we have counts for
23 papers · 1 filter
Depth Memory
David González-Martínez, Philipp Nazari, T. Konstantin Rusch +1
Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connection…
Looped Reasoning Finishes Earlier Than You Think
Sumin Park, T. Konstantin Rusch
Looped reasoning with dynamic depth can be viewed as relaxation toward solution attractors in a learned dynamical landscape. In practice, however, latent trajectories often oscilla…
Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
T. Konstantin Rusch, Tim Seyde, Jared Boyer +2
Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computat…
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4
Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in me…
Low-Pass Flow Matching
Francesco M. Ruscio, T. Konstantin Rusch
Flow Matching typically relies on white noise sources, a choice often misaligned with the power spectra of natural data, which tend to decay with frequency. To address this, we int…
On State Reduction in Linear Attention
Philipp Nazari, T. Konstantin Rusch
Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate that the hidden state of trained lin…
Quantifying Memory Use in Reinforcement Learning with Temporal Range
Rodney Lafuente-Mercado, Daniela Rus, T. Konstantin Rusch
How much does a trained RL policy actually use its past observations? We propose \emph{Temporal Range}, a model-agnostic metric that treats first-order sensitivities of multiple ve…
Neural Low-Discrepancy Sequences
Michael Etienne Van Huffel, Nathan Kirk, Makram Chahine +2
Low-discrepancy points are designed to efficiently fill the space in a uniform manner. This uniformity is highly advantageous in many problems in science and engineering, including…
The Curious Case of In-Training Compression of State Space Models
Makram Chahine, Philipp Nazari, Daniela Rus +1
State Space Models (SSMs), developed to tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference. At their core are recurrent dynamica…
Learning to Dissipate Energy in Oscillatory State-Space Models
Jared Boyer, T. Konstantin Rusch, Daniela Rus
State-space models (SSMs) are a class of networks for sequence learning that benefit from fixed state size and linear complexity with respect to sequence length, contrasting the qu…
Low Stein Discrepancy via Message-Passing Monte Carlo
Nathan Kirk, T. Konstantin Rusch, Jakob Zech +1
Message-Passing Monte Carlo (MPMC) was recently introduced as a novel low-discrepancy sampling approach leveraging tools from geometric deep learning. While originally designed for…
Relaxed Equivariance via Multitask Learning
Ahmed A. Elhag, T. Konstantin Rusch, Francesco Di Giovanni +1
Incorporating equivariance as an inductive bias into deep learning architectures to take advantage of the data symmetry has been successful in multiple applications, such as chemis…
Oscillatory State-Space Models
T. Konstantin Rusch, Daniela Rus
We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our propo…
Message-Passing Monte Carlo: Generating low-discrepancy point sets via Graph Neural Networks
T. Konstantin Rusch, Nathan Kirk, Michael M. Bronstein +2
Discrepancy is a well-known measure for the irregularity of the distribution of a point set. Point sets with small discrepancy are called low-discrepancy and are known to efficient…
How does over-squashing affect the power of GNNs?
Francesco Di Giovanni, T. Konstantin Rusch, Michael M. Bronstein +4
Graph Neural Networks (GNNs) are the state-of-the-art model for machine learning on graph-structured data. The most popular class of GNNs operate by exchanging information between…
A Survey on Oversmoothing in Graph Neural Networks
T. Konstantin Rusch, Michael M. Bronstein, Siddhartha Mishra
Node features of graph neural networks (GNNs) tend to become more similar with the increase of the network depth. This effect is known as over-smoothing, which we axiomatically def…
Multi-Scale Message Passing Neural PDE Solvers
Léonard Equer, T. Konstantin Rusch, Siddhartha Mishra
We propose a novel multi-scale message passing neural network algorithm for learning the solutions of time-dependent PDEs. Our algorithm possesses both temporal and spatial multi-s…
Gradient Gating for Deep Multi-Rate Learning on Graphs
T. Konstantin Rusch, Benjamin P. Chamberlain, Michael W. Mahoney +2
We present Gradient Gating (G), a novel framework for improving the performance of Graph Neural Networks (GNNs). Our framework is based on gating the output of GNN layers with…
Graph-Coupled Oscillator Networks
T. Konstantin Rusch, Benjamin P. Chamberlain, James Rowbottom +2
We propose Graph-Coupled Oscillator Networks (GraphCON), a novel framework for deep learning on graphs. It is based on discretizations of a second-order system of ordinary differen…
Long Expressive Memory for Sequence Modeling
T. Konstantin Rusch, Siddhartha Mishra, N. Benjamin Erichson +1
We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks wit…
UnICORNN: A recurrent model for learning very long time dependencies
T. Konstantin Rusch, Siddhartha Mishra
The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gra…
Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
T. Konstantin Rusch, Siddhartha Mishra
Circuits of biological neurons, such as in the functional parts of the brain can be modeled as networks of coupled oscillators. Inspired by the ability of these systems to express…
Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences
Siddhartha Mishra, T. Konstantin Rusch
We propose a deep supervised learning algorithm based on low-discrepancy sequences as the training set. By a combination of theoretical arguments and extensive numerical experiment…