Smaller Models, Better Generalization

Abstract

Reducing network complexity has been a major research focus in recent years with the advent of mobile technology. Convolutional Neural Networks that perform various vision tasks without memory overhaul is the need of the hour. This paper focuses on qualitative and quantitative analysis of reducing the network complexity using an upper bound on the Vapnik-Chervonenkis dimension, pruning, and quantization. We observe a general trend in improvement of accuracies as we quantize the models. We propose a novel loss function that helps in achieving considerable sparsity at comparable accuracies to that of dense models. We compare various regularizations prevalent in the literature and show the superiority of our method in achieving sparser models that generalize well.

arXiv comment10 pages, 3 figures, In Review

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Playing Atari with Deep Reinforcement Learning
• Training Very Deep Networks
• Deep Learning using Linear Support Vector Machines
• Group Sparse Regularization for Deep Neural Networks
• Neural Networks with Few Multiplications
• Memory Bounded Deep Convolutional Networks
• NoiseOut: A Simple Way to Prune Neural Networks
• Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
• The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning