Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

Shuai Yuan, Sudong Cai, Bingzhi Chen +4

Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks. In…