82 citations · 145 across the 26 of their papers we have counts for
31 papers
Depth Memory
David González-Martínez, Philipp Nazari, T. Konstantin Rusch +1
Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connection…
Looped Reasoning Finishes Earlier Than You Think
Sumin Park, T. Konstantin Rusch
Looped reasoning with dynamic depth can be viewed as relaxation toward solution attractors in a learned dynamical landscape. In practice, however, latent trajectories often oscilla…
Two-Variable Logic with Arbitrarily Many Successor Relations
Jakub Michaliszyn, Piotr Witkowski
We study two-variable first-order logic with a finite, input-dependent number of distinguished successor relations and arbitrary additional unary and binary predicates. Each distin…
Efficient Reasoning with Flow Language Models
Hanru Bai, Faissal Izermine, Oscar Davis +1
Flow Language Models (FLMs) have emerged as a continuous-state alternative to discrete diffusion language models, yet the role of their continuous representations in reasoning rema…
Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
T. Konstantin Rusch, Tim Seyde, Jared Boyer +2
Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computat…
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin +5
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by loo…
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4
Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in me…
Low-Pass Flow Matching
Francesco M. Ruscio, T. Konstantin Rusch
Flow Matching typically relies on white noise sources, a choice often misaligned with the power spectra of natural data, which tend to decay with frequency. To address this, we int…
On State Reduction in Linear Attention
Philipp Nazari, T. Konstantin Rusch
Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate that the hidden state of trained lin…
Quantifying Memory Use in Reinforcement Learning with Temporal Range
Rodney Lafuente-Mercado, Daniela Rus, T. Konstantin Rusch
How much does a trained RL policy actually use its past observations? We propose \emph{Temporal Range}, a model-agnostic metric that treats first-order sensitivities of multiple ve…
Neural Low-Discrepancy Sequences
Michael Etienne Van Huffel, Nathan Kirk, Makram Chahine +2
Low-discrepancy points are designed to efficiently fill the space in a uniform manner. This uniformity is highly advantageous in many problems in science and engineering, including…
The Curious Case of In-Training Compression of State Space Models
Makram Chahine, Philipp Nazari, Daniela Rus +1
State Space Models (SSMs), developed to tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference. At their core are recurrent dynamica…
On the optimization of discrepancy measures
François Clément, Nathan Kirk, Art B. Owen +1
Numerical optimizations can find point sets with lower discrepancies than prior methods based on algebra, and discrepancies are computationally attractive for this. The …
Learning to Move in Rhythm: Task-Conditioned Motion Policies with Orbital Stability Guarantees
Maximilian Stölzle, T. Konstantin Rusch, Zach J. Patterson +5
Learning from demonstration provides a sample-efficient approach to acquiring complex behaviors, enabling robots to move robustly, compliantly, and with fluidity. In this context,…
Learning to Dissipate Energy in Oscillatory State-Space Models
Jared Boyer, T. Konstantin Rusch, Daniela Rus
State-space models (SSMs) are a class of networks for sequence learning that benefit from fixed state size and linear complexity with respect to sequence length, contrasting the qu…
Low Stein Discrepancy via Message-Passing Monte Carlo
Nathan Kirk, T. Konstantin Rusch, Jakob Zech +1
Message-Passing Monte Carlo (MPMC) was recently introduced as a novel low-discrepancy sampling approach leveraging tools from geometric deep learning. While originally designed for…
Relaxed Equivariance via Multitask Learning
Ahmed A. Elhag, T. Konstantin Rusch, Francesco Di Giovanni +1
Incorporating equivariance as an inductive bias into deep learning architectures to take advantage of the data symmetry has been successful in multiple applications, such as chemis…
Oscillatory State-Space Models
T. Konstantin Rusch, Daniela Rus
We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our propo…
Improving Efficiency of Sampling-based Motion Planning via Message-Passing Monte Carlo
Makram Chahine, T. Konstantin Rusch, Zach J. Patterson +1
Sampling-based motion planning methods, while effective in high-dimensional spaces, often suffer from inefficiencies due to irregular sampling distributions, leading to suboptimal…
Message-Passing Monte Carlo: Generating low-discrepancy point sets via Graph Neural Networks
T. Konstantin Rusch, Nathan Kirk, Michael M. Bronstein +2
Discrepancy is a well-known measure for the irregularity of the distribution of a point set. Point sets with small discrepancy are called low-discrepancy and are known to efficient…
How does over-squashing affect the power of GNNs?
Francesco Di Giovanni, T. Konstantin Rusch, Michael M. Bronstein +4
Graph Neural Networks (GNNs) are the state-of-the-art model for machine learning on graph-structured data. The most popular class of GNNs operate by exchanging information between…
Neural Oscillators are Universal
Samuel Lanthaler, T. Konstantin Rusch, Siddhartha Mishra
Coupled oscillators are being increasingly used as the basis of machine learning (ML) architectures, for instance in sequence modeling, graph representation learning and in physica…
A Survey on Oversmoothing in Graph Neural Networks
T. Konstantin Rusch, Michael M. Bronstein, Siddhartha Mishra
Node features of graph neural networks (GNNs) tend to become more similar with the increase of the network depth. This effect is known as over-smoothing, which we axiomatically def…
Multi-Scale Message Passing Neural PDE Solvers
Léonard Equer, T. Konstantin Rusch, Siddhartha Mishra
We propose a novel multi-scale message passing neural network algorithm for learning the solutions of time-dependent PDEs. Our algorithm possesses both temporal and spatial multi-s…