Sphere Retraction Normalizations

Jie Zhang, Cheng-Fang Su, Yi-Jui Huang +1

Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing ea…