107 citations · 236 across the 32 of their papers we have counts for
3 papers · 1 filter
IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
Yuan Yin, Shashanka Venkataramanan, Tuan-Hung Vu +2
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is rando…
Multiple Choice Learning of Low-Rank Adapters for Language Modeling
Victor Letzelter, Hugo Malard, Mathieu Fontaine +4
We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference…
Spherical Perspective on Learning with Normalization Layers
Simon Roburin, Yann de Mont-Marin, Andrei Bursuc +3
Normalization Layers (NLs) are widely used in modern deep-learning architectures. Despite their apparent simplicity, their effect on optimization is not yet fully understood. This…