Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations

Guanzhi Deng, Mingyang Liu, Dapeng Wu +2

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN,…