Layer Folding: Neural Network Depth Reduction using Activation Linearization

Abstract

Despite the increasing prevalence of deep neural networks, their applicability in resource-constrained devices is limited due to their computational load. While modern devices exhibit a high level of parallelism, real-time latency is still highly dependent on networks' depth. Although recent works show that below a certain depth, the width of shallower networks must grow exponentially, we presume that neural networks typically exceed this minimal depth to accelerate convergence and incrementally increase accuracy. This motivates us to transform pre-trained deep networks that already exploit such advantages into shallower forms. We propose a method that learns whether non-linear activations can be removed, allowing to fold consecutive linear layers into one. We apply our method to networks pre-trained on CIFAR-10 and CIFAR-100 and find that they can all be transformed into shallower forms that share a similar depth. Finally, we use our method to provide more efficient alternatives to MobileNetV2 and EfficientNet-Lite architectures on the ImageNet classification task.

References in corpus (19)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Deep Networks with Stochastic Depth
• Why Deep Neural Networks for Function Approximation?
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Going in circles is the way forward: the role of recurrence in visual inference
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
• RepVGG: Making VGG-style ConvNets Great Again
• ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
• DeepReDuce: ReLU Reduction for Fast Private Inference
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• Depth Separation for Neural Networks
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• The Low-Rank Simplicity Bias in Deep Networks
• Activate or Not: Learning Customized Activation
• Compressing Deep Neural Networks via Layer Fusion
• Neural Pruning via Growing Regularization