Compressing Neural Networks with the Hashing Trick

Abstract

As deep nets are increasingly used in applications suited for mobile devices, a fundamental dilemma becomes apparent: the trend in deep learning is to grow models to absorb ever-increasing data set sizes; however mobile devices are designed with very little memory and cannot store such large models. We present a novel network architecture, HashedNets, that exploits inherent redundancy in neural networks to achieve drastic reductions in model sizes. HashedNets uses a low-cost hash function to randomly group connection weights into hash buckets, and all connections within the same hash bucket share a single parameter value. These parameters are tuned to adjust to the HashedNets weight sharing architecture with standard backprop during training. Our hashing procedure introduces no additional memory overhead, and we demonstrate on several benchmark data sets that HashedNets shrink the storage requirements of neural networks substantially while mostly preserving generalization performance.

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Practical Bayesian Optimization of Machine Learning Algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• A Reliable Effective Terascale Linear Learning System
• High-Performance Neural Networks for Visual Object Classification
• Learning Ordered Representations with Nested Dropout
• Compressed Support Vector Machines

Cited by in corpus (230)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Learning both Weights and Connections for Efficient Neural Networks
• Tensorizing Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Group Sparse Regularization for Deep Neural Networks
• Recent Advances in Convolutional Neural Networks
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
• A Survey of FPGA-Based Neural Network Accelerator
• Exploring Sparsity in Recurrent Neural Networks
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Adaptive Neural Networks for Efficient Inference
• Edge Intelligence: Architectures, Challenges, and Applications
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Sequence-Level Knowledge Distillation
• Structured Transforms for Small-Footprint Deep Learning
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• A Systematic Review on Model Watermarking for Neural Networks
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
• Network Decoupling: From Regular to Depthwise Separable Convolutions
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• A novel channel pruning method for deep neural network compression
• AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• NISP: Pruning Networks using Neuron Importance Score Propagation
• TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
• Measuring the Intrinsic Dimension of Objective Landscapes
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Automated Pruning for Deep Neural Network Compression
• Lightweight Modules for Efficient Deep Learning based Image Restoration
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• RAPIDNN: In-Memory Deep Neural Network Acceleration Framework
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• NoScope: Optimizing Neural Network Queries over Video at Scale
• On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition
• Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• BlockDrop: Dynamic Inference Paths in Residual Networks
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Low-Memory Neural Network Training: A Technical Report
• DCFNet: Deep Neural Network with Decomposed Convolutional Filters
• AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
• Accelerating Neural ODEs Using Model Order Reduction
• Quantized Convolutional Neural Networks for Mobile Devices
• Pruning Convolutional Neural Networks with Self-Supervision
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• DSConv: Efficient Convolution Operator
• Learning to Hash for Indexing Big Data - A Survey
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• Compressing Word Embeddings via Deep Compositional Code Learning
• BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Channel Compression: Rethinking Information Redundancy among Channels in CNN Architecture
• Wide Compression: Tensor Ring Nets
• Communication-Efficient Edge AI: Algorithms and Systems
• Effectiveness of Distillation Attack and Countermeasure on Neural Network Watermarking
• Word2Bits - Quantized Word Vectors
• Binary embeddings with structured hashed projections
• Provable Filter Pruning for Efficient Neural Networks
• Bolt: Accelerated Data Mining with Fast Vector Compression
• Embedding Deep Networks into Visual Explanations
• LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Lightweight Residual Densely Connected Convolutional Neural Network
• Learning Compact Recurrent Neural Networks
• Resource-Efficient Neural Networks for Embedded Systems
• Rate Distortion For Model Compression: From Theory To Practice
• ACDC: A Structured Efficient Linear Layer
• Binarized Neural Networks on the ImageNet Classification Task
• Weight-dependent Gates for Network Pruning
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Fast ConvNets Using Group-wise Brain Damage
• Deep Fried Convnets
• Iteratively Training Look-Up Tables for Network Quantization
• Knowledge Distillation for End-to-End Person Search
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Continual 3D Convolutional Neural Networks for Real-time Processing of Videos
• Exploiting Errors for Efficiency: A Survey from Circuits to Algorithms
• Distilling Word Embeddings: An Encoding Approach
• Convolutional Neural Network Pruning with Structural Redundancy Reduction
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• A Highly Parallel FPGA Implementation of Sparse Neural Network Training
• Teacher-Student Compression with Generative Adversarial Networks
• Trained Rank Pruning for Efficient Deep Neural Networks
• On the energy landscape of deep networks
• OpenEI: An Open Framework for Edge Intelligence
• Benchmarking Quantized Neural Networks on FPGAs with FINN
• RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks
• LCNN: Lookup-based Convolutional Neural Network
• Intermittent Learning: On-Device Machine Learning on Intermittently Powered System
• Evolution in Groups: A deeper look at synaptic cluster driven evolution of deep neural networks
• Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems
• HRank: Filter Pruning using High-Rank Feature Map
• AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
• Full deep neural network training on a pruned weight budget
• Rethinking Class-Discrimination Based CNN Channel Pruning
• ApproxNet: Content and Contention-Aware Video Analytics System for Embedded Clients
• Structured Convolution Matrices for Energy-efficient Deep learning
• Kronecker Recurrent Units
• Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
• Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters
• Super Resolution Using Segmentation-Prior Self-Attention Generative Adversarial Network
• Scalable Model Compression by Entropy Penalized Reparameterization
• Text classification with word embedding regularization and soft similarity measure
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Meta Feature Modulator for Long-tailed Recognition
• Interleaver Design for Deep Neural Networks
• Learning Filter Basis for Convolutional Neural Network Compression
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Rethinking Numerical Representations for Deep Neural Networks
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion
• Understanding Sparse JL for Feature Hashing
• Shapley Value as Principled Metric for Structured Network Pruning
• Data-Free Learning of Student Networks
• Harmonic Networks: Integrating Spectral Information into CNNs
• CDFI: Compression-Driven Network Design for Frame Interpolation
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Dynamic Runtime Feature Map Pruning
• Multigrid Neural Architectures
• Dependency Aware Filter Pruning
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error
• BitNet: Bit-Regularized Deep Neural Networks
• Sparsely Aggregated Convolutional Networks
• Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds
• Adaptive Selection of Deep Learning Models on Embedded Systems
• Saec: Similarity-Aware Embedding Compression in Recommendation Systems
• COPS: Controlled Pruning Before Training Starts
• Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Self-Supervised GAN Compression
• On Hiding Neural Networks Inside Neural Networks
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Understanding and Training Deep Diagonal Circulant Neural Networks
• Predicting Terrain Mechanical Properties in Sight for Planetary Rovers with Semantic Clues
• PIMBALL: Binary Neural Networks in Spintronic Memory
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• Accelerate CNN via Recursive Bayesian Pruning
• Distilling portable Generative Adversarial Networks for Image Translation
• DCT-Conv: Coding filters in convolutional networks with Discrete Cosine Transform
• SECS: Efficient Deep Stream Processing via Class Skew Dichotomy
• Collaborative Learning for Extremely Low Bit Asymmetric Hashing
• Learning Compressed Transforms with Low Displacement Rank
• Logically Synthesized, Hardware-Accelerated, Restricted Boltzmann Machines for Combinatorial Optimization and Integer Factorization
• Parameter Efficient Deep Neural Networks with Bilinear Projections
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• FedOCR: Communication-Efficient Federated Learning for Scene Text Recognition
• Compression of Deep Neural Networks on the Fly
• DNQ: Dynamic Network Quantization
• FALCON: Lightweight and Accurate Convolution
• Separable Layers Enable Structured Efficient Linear Substitutions
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Robust Membership Encoding: Inference Attacks and Copyright Protection for Deep Learning
• Balanced Binary Neural Networks with Gated Residual
• Reinforcement Learning with Chromatic Networks for Compact Architecture Search
• Stochastic Model Pruning via Weight Dropping Away and Back
• Extreme Classification in Log Memory
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Massively Parallel Video Networks
• Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference
• Data Analytics Service Composition and Deployment on Edge Devices
• DeepThin: A Self-Compressing Library for Deep Neural Networks
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
• ESPN: Extremely Sparse Pruned Networks
• Generative Design of Hardware-aware DNNs
• Quantized Epoch-SGD for Communication-Efficient Distributed Learning
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• BasisConv: A method for compressed representation and learning in CNNs
• Compressing Deep Neural Networks: A New Hashing Pipeline Using Kac's Random Walk Matrices
• FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
• Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks
• Deep Asymmetric Hashing with Dual Semantic Regression and Class Structure Quantization
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks
• Robust Student Network Learning
• Structured Sparsification with Joint Optimization of Group Convolution and Channel Shuffle
• Composite Binary Decomposition Networks
• Efficient and Robust Machine Learning for Real-World Systems
• Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• Improving Network Slimming with Nonconvex Regularization
• DP-Net: Dynamic Programming Guided Deep Neural Network Compression
• Revisiting hard thresholding for DNN pruning
• A Survey on GAN Acceleration Using Memory Compression Technique
• Learning Multi-granular Quantized Embeddings for Large-Vocab Categorical Features in Recommender Systems
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Learning SMaLL Predictors
• Identifying and Exploiting Structures for Reliable Deep Learning
• Block-term Tensor Neural Networks
• Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression
• Backpropagating through Fréchet Inception Distance
• Attn-HybridNet: Improving Discriminability of Hybrid Features with Attention Fusion
• Continual learning using hash-routed convolutional neural networks
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications
• Hybrid Cosine Based Convolutional Neural Networks
• Virtual Experience to Real World Application: Sidewalk Obstacle Avoidance Using Reinforcement Learning for Visually Impaired
• Differentiable Architecture Pruning for Transfer Learning
• Shakeout: A New Approach to Regularized Deep Neural Network Training
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
• A Comparative Study of Neural Network Compression
• Radius Adaptive Convolutional Neural Network
• CountSketches, Feature Hashing and the Median of Three
• Stochastic Computing for Hardware Implementation of Binarized Neural Networks
• All Word Embeddings from One Embedding
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• Streaming Binary Sketching based on Subspace Tracking and Diagonal Uniformization
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces
• Accuracy to Throughput Trade-offs for Reduced Precision Neural Networks on Reconfigurable Logic
• Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation
• An Overview of Datatype Quantization Techniques for Convolutional Neural Networks
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• Training compact deep learning models for video classification using circulant matrices
• An Improving Framework of regularization for Network Compression
• Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
• Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices