5 papers
Compact Memory for Continual Logistic Regression
Yohan Jung, Hyungi Lee, Wenlong Chen +4
Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential p…
Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
Navish Kumar, Thomas Möllenhoff, Mohammad Emtiyaz Khan +1
Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is…
Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks
Keigo Nishida, Eren Mehmet Kıral, Kenichi Bannai +2
Studies in neuroscience have shown that biological synapses follow a log-normal distribution whose transitioning can be explained by noisy multiplicative dynamics. Biological netwo…
Improving LoRA with Variational Learning
Bai Cong, Nico Daheim, Yuesong Shen +3
Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can som…
Variational Learning Finds Flatter Solutions at the Edge of Stability
Avrajit Ghosh, Bai Cong, Rio Yokota +5
Variational Learning (VL) has recently gained popularity for training deep neural networks. Part of its empirical success can be explained by theories such as PAC-Bayes bounds, min…