Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

Abstract

This work presents deep asymmetric networks with a set of node-wise variant activation functions. The nodes' sensitivities are affected by activation function selections such that the nodes with smaller indices become increasingly more sensitive. As a result, features learned by the nodes are sorted by the node indices in the order of their importance. Asymmetric networks not only learn input features but also the importance of those features. Nodes of lesser importance in asymmetric networks can be pruned to reduce the complexity of the networks, and the pruned networks can be retrained without incurring performance losses. We validate the feature-sorting property using both shallow and deep asymmetric networks as well as deep asymmetric networks transferred from famous networks.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• An Entropy-based Pruning Method for CNN Compression
• Memory Bounded Deep Convolutional Networks
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
• Regularizing CNNs with Locally Constrained Decorrelations
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• Activation Ensembles for Deep Neural Networks

Cited by in corpus (1)

• Efficient architecture for deep neural networks with heterogeneous sensitivity