NOBLE: Accelerating Transformers with Nonlinear Low-Rank Branches

Ethan Smith

We introduce NOBLE (Nonlinear lOw-rank Branch for Linear Enhancement), an architectural augmentation that adds nonlinear low-rank branches to transformer linear layers. Unlike LoRA…