Deep Isometric Learning for Visual Recognition

Abstract

Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.

ICML 2020; Code: https://github.com/HaozhiQi/ISONet; Website: https://haozhiqi.github.io/ISONet

Cited by in corpus (15)

• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• What causes the test error? Going beyond bias-variance via ANOVA
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Orthogonal Over-Parameterized Training
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Batch Normalization with Enhanced Linear Transformation
• Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold
• Learning with Hyperspherical Uniformity
• PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python