Deep Fried Convnets
arXiv:1412.7149
Abstract
The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important for operating deep neural networks in memory constrained environments such as GPUs or embedded devices. In this paper we show how kernel methods, in particular a single Fastfood layer, can be used to replace all fully connected layers in a deep convolutional neural network. This novel Fastfood layer is also end-to-end trainable in conjunction with convolutional layers, allowing us to combine them into a new architecture, named deep fried convolutional networks, which substantially reduces the memory footprint of convolutional networks trained on MNIST and ImageNet with no drop in predictive performance.
arXiv commentsvd experiments included
References in corpus (14)
- Distilling the Knowledge in a Neural Network
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Going Deeper with Convolutions
- Weight Uncertainty in Neural Networks
- Compressing Deep Convolutional Networks using Vector Quantization
- One weird trick for parallelizing convolutional neural networks
- Predicting Parameters in Deep Learning
- Compressing Neural Networks with the Hashing Trick
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Memory Bounded Deep Convolutional Networks
- Scalable Kernel Methods via Doubly Stochastic Gradients
- A la Carte - Learning Fast Kernels
- Speeding Up Neural Networks for Large Scale Classification using WTA Hashing
Cited by in corpus (14)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Learning both Weights and Connections for Efficient Neural Networks
- Scalable Kernel Methods via Doubly Stochastic Gradients
- Diversity Networks: Neural Network Compression Using Determinantal Point Processes
- An exploration of parameter redundancy in deep networks with circulant projections
- Compact Bilinear Pooling
- Compressing Convolutional Neural Networks
- Quantized Convolutional Neural Networks for Mobile Devices
- A Kronecker-factored approximate Fisher matrix for convolution layers
- Deep Kernel Learning
- Training Sparse Neural Networks
- On Binary Embedding using Circulant Matrices
- Recurrent Residual Module for Fast Inference in Videos