oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

Haoqiang Guo, Xuyi Chen, Bo Ke +5

Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n\times n$ residual matrix. Leaving t…