82 citations · 228 across the 42 of their papers we have counts for
48 papers
What "Form'' of Uncertainty Does Conformal Prediction Quantify?
Soroush H. Zargarbashi, Mohammad Sadegh Akhondzadeh, Rebekka Burkholz +1
Conformal prediction (CP) is a post-hoc calibration method that returns prediction sets with distribution-free coverage guarantees, and the size of the output set is often interpre…
From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon
Tom Jacobs, Rebekka Burkholz
Simplicity bias, which can arise from sequential feature learning during training, has been proposed as a general explanation for the success of deep learning. An important driver…
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz
Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoS…
Mind the Drift: Diagonal Linear Networks Under Large Learning Rates
Aniket Sanyal, Tom Jacobs, Rebekka Burkholz
Large learning rates can qualitatively change the trajectory of neural network training, often pushing optimization into regimes far from classical gradient-flow behavior. The Edge…
Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?
Franka Bause, Jonas Niederle, Martin Pawelczyk +1
The effectiveness of multi-agent LLM deliberation depends not only on the agents' individual predictions, but also on how they communicate and collaborate. We study this mechanism…
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
Mohammed Adnan, Rohan Jain, Tom Jacobs +4
Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST…
HORST: Composing Optimizer Geometries for Sparse Transformer Training
Tom Jacobs, Rohan Jain, Rebekka Burkholz
Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optim…
Don't Trust Stubborn Neighbors: A Security Framework for Agentic Networks
Samira Abedini, Sina Mavali, Lea Schönherr +2
Large Language Model (LLM)-based Multi-Agent Systems (MASs) are increasingly deployed for agentic tasks, such as web automation, itinerary planning, and collaborative problem solvi…
Bridging Domains through Subspace-Aware Model Merging
Levy Chaves, Chao Zhou, Rebekka Burkholz +2
Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or mul…
Never Saddle for Reparameterized Steepest Descent as Mirror Flow
Tom Jacobs, Chao Zhou, Rebekka Burkholz
How does the choice of optimization algorithm shape a model's ability to learn features? To address this question for steepest descent methods --including sign descent, which is cl…
Frequency-Based Hyperparameter Selection in Games
Aniket Sanyal, Baraah A. M. Sidahmed, Rebekka Burkholz +1
Learning in smooth games fundamentally differs from standard minimization due to rotational dynamics, which invalidate classical hyperparameter tuning strategies. Despite their pra…
Fixed Aggregation Features Can Rival GNNs
Celia Rubio-Madrigal, Rebekka Burkholz
Graph neural networks (GNNs) are widely believed to excel at node representation learning through trainable neighborhood aggregations. We challenge this view by introducing Fixed A…
Robustness of Mixtures of Experts to Feature Noise
Dong Sun, Rahul Nittala, Rebekka Burkholz
Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regi…
The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis
Hoang Pham, The-Anh Ta, Tom Jacobs +2
Sparse neural networks promise efficiency, yet training them effectively remains a fundamental challenge. Despite advances in pruning methods that create sparse architectures, unde…
Pay Attention to Small Weights
Chao Zhou, Tom Jacobs, Advait Gadhikar +1
Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict tr…
Hyperbolic Aware Minimization: Implicit Bias for Sparsity
Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal +1
Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise o…
When Shift Happens - Confounding Is to Blame
Abbavaram Gowtham Reddy, Celia Rubio-Madrigal, Rebekka Burkholz +1
Distribution shifts introduce uncertainty that undermines the robustness and generalization capabilities of machine learning models. While conventional wisdom suggests that learnin…
Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Tom Jacobs, Chao Zhou, Rebekka Burkholz
Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent…
Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
Advait Gadhikar, Tom Jacobs, Chao Zhou +1
The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the…
GNNs Getting ComFy: Community and Feature Similarity Guided Rewiring
Celia Rubio-Madrigal, Adarsh Jamadandi, Rebekka Burkholz
Maximizing the spectral gap through graph rewiring has been proposed to enhance the performance of message-passing graph neural networks (GNNs) by addressing over-squashing. Howeve…
Mask in the Mirror: Implicit Sparsification
Tom Jacobs, Rebekka Burkholz
Continuous sparsification strategies are among the most effective methods for reducing the inference costs and memory demands of large-scale neural networks. A key factor in their…
GATE: How to Keep Out Intrusive Neighbors
Nimrah Mustafa, Rebekka Burkholz
Graph Attention Networks (GATs) are designed to provide flexible neighborhood aggregation that assigns weights to neighbors according to their importance. In practice, however, GAT…
Cyclic Sparse Training: Is it Enough?
Advait Gadhikar, Sree Harsha Nelaturu, Rebekka Burkholz
The success of iterative pruning methods in achieving state-of-the-art sparse networks has largely been attributed to improved mask identification and an implicit regularization in…
CAVACHON: a hierarchical variational autoencoder to integrate multi-modal single-cell data
Ping-Han Hsieh, Ru-Xiu Hsiao, Katalin Ferenc +6
Paired single-cell sequencing technologies enable the simultaneous measurement of complementary modalities of molecular data at single-cell resolution. Along with the advances in t…