Low-Rank Attention Residuals

Jonathan Su

Attention Residuals replace the fixed residual sum with depthwise attention over previous sub-layer outputs in large language models (LLMs), but use each output as both a full-dime…