The Early Phase of Neural Network Training

Abstract

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here, we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this behavior, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are not inherently label-dependent, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

arXiv commentICLR 2020 Camera Ready. Available on OpenReview at https://openreview.net/forum?id=Hkl1iRNFwS

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Wide Residual Networks
• Rethinking the Value of Network Pruning
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Stabilizing the Lottery Ticket Hypothesis
• Gradient Descent Happens in a Tiny Subspace
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Critical Learning Periods in Deep Neural Networks

Cited by in corpus (25)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• What Do Neural Networks Learn When Trained With Random Labels?
• Consensus Control for Decentralized Deep Learning
• Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
• GANs Can Play Lottery Tickets Too
• Diet deep generative audio models with structured lottery
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• When to Prune? A Policy towards Early Structural Pruning
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Intraclass clustering: an implicit learning ability that regularizes DNNs
• Ultra-light deep MIR by trimming lottery tickets
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Effect of the initial configuration of weights on the training and function of artificial neural networks
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?