TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

Yuxuan Gu, Wuyang Zhou, Huijun Xing +1

Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by…