Practical recommendations for gradient-based training of deep architectures

Abstract

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

References in corpus (3)

• Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery
• From Machine Learning to Machine Reasoning
• Implicit Density Estimation by Local Moment Matching to Sample from Auto-Encoders

Cited by in corpus (63)

• A Survey on Deep Learning in Medical Image Analysis
• Methods for Interpreting and Understanding Deep Neural Networks
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks
• Systematic evaluation of CNN advances on the ImageNet
• Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
• Segmentation of Glioma Tumors in Brain Using Deep Convolutional Neural Network
• Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• How Does Learning Rate Decay Help Modern Neural Networks?
• Representation Learning: A Review and New Perspectives
• An Empirical Exploration of Curriculum Learning for Neural Machine Translation
• Deep Networks as Approximators of Optimal Transfers Solutions in Multitarget Missions
• A machine learning approach for efficient uncertainty quantification using multiscale methods
• A Low Effort Approach to Structured CNN Design Using PCA
• Online Learning Rate Adaptation with Hypergradient Descent
• On the Potential of Simple Framewise Approaches to Piano Transcription
• Photometry of high-redshift blended galaxies using deep learning
• A Deep Learning Approach for Forecasting Air Pollution in South Korea Using LSTM
• uTHCD: A New Benchmarking for Tamil Handwritten OCR
• Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer's Disease using structural MR and FDG-PET images
• Knock-Knock: Acoustic Object Recognition by using Stacked Denoising Autoencoders
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
• 'CADSketchNet' -- An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks
• Representing high throughput expression profiles via perturbation barcodes reveals compound targets
• Training Neural Networks with Stochastic Hessian-Free Optimization
• Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification
• A Framework for Parallel and Distributed Training of Neural Networks
• Factors of Transferability for a Generic ConvNet Representation
• On gray-box modeling for virtual flow metering
• Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
• Percival: Making In-Browser Perceptual Ad Blocking Practical With Deep Learning
• One Vector is Not Enough: Entity-Augmented Distributional Semantics for Discourse Relations
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• Exploring Context with Deep Structured models for Semantic Segmentation
• Improved Part Segmentation Performance by Optimising Realism of Synthetic Images using Cycle Generative Adversarial Networks
• Improving Recurrent Neural Networks For Sequence Labelling
• AdaS: Adaptive Scheduling of Stochastic Gradients
• Neural Network Models for Implicit Discourse Relation Classification in English and Chinese without Surface Features
• Distributed Training Large-Scale Deep Architectures
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Deep Learning in Information Security
• Training Deep Networks without Learning Rates Through Coin Betting
• Cells in Multidimensional Recurrent Neural Networks
• Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification
• Learning Strict Identity Mappings in Deep Residual Networks
• Tricks from Deep Learning
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• A Generalizable Approach to Learning Optimizers
• Spatial Mapping with Gaussian Processes and Nonstationary Fourier Features
• Design of Capacity-Approaching Low-Density Parity-Check Codes using Recurrent Neural Networks
• Neural Word Segmentation with Rich Pretraining
• Effective Feature Learning with Unsupervised Learning for Improving the Predictive Models in Massive Open Online Courses
• Custom Dual Transportation Mode Detection by Smartphone Devices Exploiting Sensor Diversity
• Understanding and Optimizing Packed Neural Network Training for Hyper-Parameter Tuning
• Collaborative Filtering using Denoising Auto-Encoders for Market Basket Data
• The observer-assisted method for adjusting hyper-parameters in deep learning algorithms
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Network Parameter Learning Using Nonlinear Transforms, Local Representation Goals and Local Propagation Constraints
• A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
• Short-term Cognitive Networks, Flexible Reasoning and Nonsynaptic Learning
• One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks
• Sequenced-Replacement Sampling for Deep Learning