Predicting Parameters in Deep Learning

Abstract

We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• High-Performance Neural Networks for Visual Object Classification
• Deep Learning of Representations: Looking Forward
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields

Cited by in corpus (114)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• LoRA: Low-Rank Adaptation of Large Language Models
• Compressing Deep Convolutional Networks using Vector Quantization
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• The Loss Surfaces of Multilayer Networks
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Group Sparse Regularization for Deep Neural Networks
• Bayesian Compression for Deep Learning
• Recent Advances in Convolutional Neural Networks
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• Infinite Feature Selection: A Graph-based Feature Filtering Approach
• Learning feed-forward one-shot learners
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• A Survey on Methods and Theories of Quantized Neural Networks
• An Entropy-based Pruning Method for CNN Compression
• Compressing Neural Networks using the Variational Information Bottleneck
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Compressing Recurrent Neural Network with Tensor Train
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Ultimate tensorization: compressing convolutional and FC layers alike
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Deep Polynomial Neural Networks
• A Low Effort Approach to Structured CNN Design Using PCA
• DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• NISP: Pruning Networks using Neuron Importance Score Propagation
• Measuring the Intrinsic Dimension of Objective Landscapes
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Automated Pruning for Deep Neural Network Compression
• On the Role of ViT and CNN in Semantic Communications: Analysis and Prototype Validation
• Low-complexity Approximate Convolutional Neural Networks
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Hybrid Tensor Decomposition in Neural Network Compression
• On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections
• PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Memory Matching Networks for One-Shot Image Recognition
• DecomposeMe: Simplifying ConvNets for End-to-End Learning
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Quantized Convolutional Neural Networks for Mobile Devices
• Compressing Convolutional Neural Networks
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Learning to Prune Filters in Convolutional Neural Networks
• Channel Compression: Rethinking Information Redundancy among Channels in CNN Architecture
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Convolution by Evolution: Differentiable Pattern Producing Networks
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• Distilled Neural Networks for Efficient Learning to Rank
• Lightweight Residual Densely Connected Convolutional Neural Network
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Stable Tensor Neural Networks for Rapid Deep Learning
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• ACDC: A Structured Efficient Linear Layer
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• Deep Fried Convnets
• Autonomous Deep Learning: Incremental Learning of Denoising Autoencoder for Evolving Data Streams
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• A Highly Parallel FPGA Implementation of Sparse Neural Network Training
• TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
• LCNN: Lookup-based Convolutional Neural Network
• Learning Two Layer Rectified Neural Networks in Polynomial Time
• Coordinating Filters for Faster Deep Neural Networks
• Training Sparse Neural Networks
• Generating Neural Networks with Neural Networks
• DeepFont: Identify Your Font from An Image
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• Blending LSTMs into CNNs
• Harmonic Networks: Integrating Spectral Information into CNNs
• Consistent Sparse Deep Learning: Theory and Computation
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• Which *BERT? A Survey Organizing Contextualized Encoders
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Compressing CNN Kernels for Videos Using Tucker Decompositions: Towards Lightweight CNN Applications
• Neural Network Regularization via Robust Weight Factorization
• A Generalized Lottery Ticket Hypothesis
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Compressing LSTM Networks by Matrix Product Operators
• Knowledge Distillation via Instance-level Sequence Learning
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• Compression of Deep Neural Networks on the Fly
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Recent Advances in Convolutional Neural Network Acceleration
• Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization
• Efficient and Robust Machine Learning for Real-World Systems
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Multi-objective Evolutionary Approach for Efficient Kernel Size and Shape for CNN
• Accuracy to Throughput Trade-offs for Reduced Precision Neural Networks on Reconfigurable Logic
• Sparse Architectures for Text-Independent Speaker Verification Using Deep Neural Networks
• Training compact deep learning models for video classification using circulant matrices
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
• Fixed-point Factorized Networks
• Expressive power of outer product manifolds on feed-forward neural networks