TensLoRA: Tensor Alternatives for Low-Rank Adaptation

Axel Marmoret, Reda Bensaid, Jonathan Lys +2

Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are cons…