Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts

Abstract

We analyze architectural features of Deep Neural Networks (DNNs) using the so-called Neural Tangent Kernel (NTK), which describes the training and generalization of DNNs in the infinite-width setting. In this setting, we show that for fully-connected DNNs, as the depth grows, two regimes appear: "order", where the (scaled) NTK converges to a constant, and "chaos", where it converges to a Kronecker delta. Extreme order slows down training while extreme chaos hinders generalization. Using the scaled ReLU as a nonlinearity, we end up in the ordered regime. In contrast, Layer Normalization brings the network into the chaotic regime. We observe a similar effect for Batch Normalization (BN) applied after the last nonlinearity. We uncover the same order and chaos modes in Deep Deconvolutional Networks (DC-NNs). Our analysis explains the appearance of so-called checkerboard patterns and border artifacts. Moving the network into the chaotic regime prevents checkerboard patterns; we propose a graph-based parametrization which eliminates border artifacts; finally, we introduce a new layer-dependent learning rate to improve the convergence of DC-NNs. We illustrate our findings on DCGANs: the ordered regime leads to a collapse of the generator to a checkerboard mode, which can be avoided by tuning the nonlinearity to reach the chaotic regime. As a result, we are able to obtain good quality samples for DCGANs without BN.

References in corpus (15)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A guide to convolution arithmetic for deep learning
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• A Mean Field Theory of Batch Normalization
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Cited by in corpus (10)

• High-Performance Large-Scale Image Recognition Without Normalization
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• On the infinite width limit of neural networks with a standard parameterization
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Implicit bias of deep linear networks in the large learning rate phase
• DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning