JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

Biswa Sengupta, Jinhua Wang, Leo Brunswic

Recent advances in deep learning, exemplified by Hyper-Connections (HC), have expanded the residual connection paradigm by introducing wider residual streams and diverse connectivi…