Deep learning for pedestrians: backpropagation in CNNs
arXiv:1811.11987
Abstract
The goal of this document is to provide a pedagogical introduction to the main concepts underpinning the training of deep neural networks using gradient descent; a process known as backpropagation. Although we focus on a very influential class of architectures called "convolutional neural networks" (CNNs) the approach is generic and useful to the machine learning community as a whole. Motivated by the observation that derivations of backpropagation are often obscured by clumsy index-heavy narratives that appear somewhat mathemagical, we aim to offer a conceptually clear, vectorized description that articulates well the higher level logic. Following the principle of "writing is nature's way of letting you know how sloppy your thinking is", we try to make the calculations meticulous, self-contained and yet as intuitive as possible. Taking nothing for granted, ample illustrations serve as visual guides and an extensive bibliography is provided for further explorations. (For the sake of clarity, long mathematical derivations and visualizations have been broken up into short "summarized views" and longer "detailed views" encoded into the PDF as optional content groups. Some figures contain animations designed to illustrate important concepts in a more engaging style. For these reasons, we advise to download the document locally and open it using Adobe Acrobat Reader. Other viewers were not tested and may not render the detailed views, animations correctly.)
References in corpus (18)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- An overview of gradient descent optimization algorithms
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- DARTS: Differentiable Architecture Search
- A guide to convolution arithmetic for deep learning
- cuDNN: Efficient Primitives for Deep Learning
- Revisiting Small Batch Training for Deep Neural Networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Measuring the tendency of CNNs to Learn Surface Statistical Regularities
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- Why do deep convolutional networks generalize so poorly to small image transformations?
- Do CIFAR-10 Classifiers Generalize to CIFAR-10?
- Norm matters: efficient and accurate normalization schemes in deep networks
- Innateness, AlphaZero, and Artificial Intelligence
- Stable Recurrent Models
- Neural Arithmetic Logic Units
- Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification