5 papers
FOGO: Forgetting-aware Orthogonalization Optimizer
Toan Nguyen, Yang Liu, Trung Le +2
We argue that forgetting is not confined to continual learning but is a general optimization phenomenon: during standard training, dominant mini-batch gradients suppress rare but u…
RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
Tuan Truong, Chau Nguyen, Huy Nguyen +3
Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remaine…
Improving Generalization with Flat Hilbert Bayesian Inference
Tuan Truong, Quyen Tran, Quan Pham-Ngoc +3
We introduce Flat Hilbert Bayesian Inference (FHBI), an algorithm designed to enhance generalization in Bayesian inference. Our approach involves an iterative two-step procedure wi…
Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
Minh Le, Chau Nguyen, Huy Nguyen +3
Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adop…
Leveraging Hierarchical Taxonomies in Prompt-based Continual Learning
Quyen Tran, Hoang Phan, Minh Le +6
Humans perceive the world as a series of sequential events, which can be hierarchically organized with different levels of abstraction based on conceptual knowledge. Drawing inspir…