8 citations · 24 across the 28 of their papers we have counts for
34 papers
A Generalization of Amari's Bayesian Duality
Mohammad Emtiyaz Khan, Thomas Möllenhoff
Amari's contributions to information geometry and machine learning are well known. Here, we revisit Amari's work on Bayesian duality which has not received as much attention. We co…
Guiding Posterior Exploration with Optimizer-Derived Geometry
Moritz Schlager, Emanuel Sommer, Thomas Möllenhoff +1
Sampling-based methods offer a principled approach to uncertainty quantification in Bayesian neural networks. Their practical use, however, is often challenged by the computational…
SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks
Adrian Robert Minut, Nico Daheim, Marco Miani +3
Structured weight-uncertainty can improve many aspects of deep learning, but it remains costly to estimate and difficult to implement. Here, we show that these issues can be addres…
Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning
Tobias Jan Wieczorek, Leon de Andrade, Thomas Möllenhoff +1
Modern deep learning models remain notoriously prone to overconfidence, limiting their reliability in high-stakes applications. Bayesian methods aim to counter this by learning a d…
Joint Model and Data Sparsification via the Marginal Likelihood
Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth +2
Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic rel…
Position: agentic AI orchestration should be Bayes-consistent
Theodore Papamarkou, Pierre Alquier, Matthias Bauer +27
LLMs excel at predictive tasks and complex reasoning tasks, but many high-value deployments rely on decisions under uncertainty, for example, which tool to call, which expert to co…
A Stein Identity for q-Gaussians with Bounded Support
Sophia Sklaviadis, Thomas Moellenhoff, Andre F. T. Martins +2
Stein's identity is a fundamental tool in machine learning with applications in generative models, stochastic optimization, and other problems involving gradients of expectations u…
Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
Navish Kumar, Thomas Möllenhoff, Mohammad Emtiyaz Khan +1
Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is…
Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks
Keigo Nishida, Eren Mehmet Kıral, Kenichi Bannai +2
Studies in neuroscience have shown that biological synapses follow a log-normal distribution whose transitioning can be explained by noisy multiplicative dynamics. Biological netwo…
Improving LoRA with Variational Learning
Bai Cong, Nico Daheim, Yuesong Shen +3
Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can som…
Federated ADMM from Bayesian Duality
Thomas Möllenhoff, Siddharth Swaroop, Finale Doshi-Velez +1
We propose a new Bayesian approach to generalize the federated Alternating Direction Method of Multipliers (ADMM). We show that the solutions of variational-Bayesian (VB) objective…
Variational Learning Finds Flatter Solutions at the Edge of Stability
Avrajit Ghosh, Bai Cong, Rio Yokota +5
Variational Learning (VL) has recently gained popularity for training deep neural networks. Part of its empirical success can be explained by theories such as PAC-Bayes bounds, min…
Uncertainty-Aware Decoding with Minimum Bayes Risk
Nico Daheim, Clara Meister, Thomas Möllenhoff +1
Despite their outstanding performance in the majority of scenarios, contemporary language models still occasionally generate undesirable outputs, for example, hallucinated text. Wh…
Natural Variational Annealing for Multimodal Optimization
Tâm LeMinh, Julyan Arbel, Thomas Möllenhoff +2
We introduce a new multimodal optimization approach called Natural Variational Annealing (NVA) that combines the strengths of three foundational concepts to simultaneously search f…
Variational Model Merging for Pareto Front Estimation in Multitask Finetuning
Hugo Monzón Maldonado, Nico Daheim, Thomas Möllenhoff +2
Pareto fronts are useful to find good task-mixing strategies for multitask finetuning, but they are also costly to compute. To reduce costs, recent works have used existing model m…
Variational Low-Rank Adaptation Using IVON
Bai Cong, Nico Daheim, Yuesong Shen +4
We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW…
Conformal Prediction via Regression-as-Classification
Etash Guha, Shlok Natarajan, Thomas Möllenhoff +2
Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed…
Variational Learning is Effective for Large Deep Networks
Yuesong Shen, Nico Daheim, Bai Cong +8
We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variati…
The Memory Perturbation Equation: Understanding Model's Sensitivity to Data
Peter Nickl, Lu Xu, Dharmesh Tailor +2
Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory…
Model Merging by Uncertainty-Based Gradient Matching
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti +2
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weight…
The Lie-Group Bayesian Learning Rule
Eren Mehmet Kıral, Thomas Möllenhoff, Mohammad Emtiyaz Khan
The Bayesian Learning Rule provides a framework for generic algorithm design but can be difficult to use for three reasons. First, it requires a specific parameterization of expone…
SAM as an Optimal Relaxation of Bayes
Thomas Möllenhoff, Mohammad Emtiyaz Khan
Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understoo…
Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields
Hartmut Bauermeister, Emanuel Laude, Thomas Möllenhoff +2
Dual decomposition approaches in nonconvex optimization may suffer from a duality gap. This poses a challenge when applying them directly to nonconvex problems such as MAP-inferenc…
Optimization of Graph Total Variation via Active-Set-based Combinatorial Reconditioning
Zhenzhang Ye, Thomas Möllenhoff, Tao Wu +1
Structured convex optimization on weighted graphs finds numerous applications in machine learning and computer vision. In this work, we propose a novel adaptive preconditioning str…