FreezeOut: Accelerate Training by Progressively Freezing Layers

Abstract

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut

Extended Abstract

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Deep Networks with Stochastic Depth

Cited by in corpus (8)

• RoboCup 2019 AdultSize Winner NimbRo: Deep Learning Perception, In-Walk Kick, Push Recovery, and Team Play Capabilities
• Neural Network Retraining for Model Serving
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Deep Networks with Fast Retraining
• Parameter Prediction for Unseen Deep Architectures
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• SuperNet -- An efficient method of neural networks ensembling
• NimbRo Robots Winning RoboCup 2018 Humanoid AdultSize Soccer Competitions