On the Origin of Deep Learning

Abstract

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

70 pages, 200 references

References in corpus (33)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• LSTM: A Search Space Odyssey
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Wide Residual Networks
• Tutorial on Variational Autoencoders
• Improved Techniques for Training GANs
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Understanding deep learning requires rethinking generalization
• An Analysis of Deep Neural Network Models for Practical Applications
• Energy-based Generative Adversarial Network
• Deep Convolutional Inverse Graphics Network
• Multiple Object Recognition with Visual Attention
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Residual Networks of Residual Networks: Multilevel Residual Networks
• How to Construct Deep Recurrent Neural Networks
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• The Power of Depth for Feedforward Neural Networks
• Identity Mappings in Deep Residual Networks
• Harnessing Deep Neural Networks with Logic Rules
• Multi-Task Cross-Lingual Sequence Tagging from Scratch
• On the saddle point problem for non-convex optimization
• Reweighted Wake-Sleep
• Calibrating Energy-based Generative Adversarial Networks
• Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
• Dropout with Expectation-linear Regularization
• Understanding Dropout: Training Multi-Layer Perceptrons with Auxiliary Independent Stochastic Neurons

Cited by in corpus (24)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Overcoming catastrophic forgetting with hard attention to the task
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Recent Advances in Deep Learning: An Overview
• Trends in deep learning for medical hyperspectral image analysis
• Deep Learning for Genomics: A Concise Overview
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
• Encoding Candlesticks as Images for Patterns Classification Using Convolutional Neural Networks
• COVID-19 Diagnosis from Cough Acoustics using ConvNets and Data Augmentation
• Towards Explainability of Machine Learning Models in Insurance Pricing
• Accelerating Deterministic and Stochastic Binarized Neural Networks on FPGAs Using OpenCL
• Continual Learning with Deep Artificial Neurons
• Predict Forex Trend via Convolutional Neural Networks
• Parkinson's Disease Diagnosis Using Deep Learning
• Evolving Neural Networks in Reinforcement Learning by means of UMDAc
• Deep Octonion Networks
• Deep Learning Paradigm with Transformed Monolingual Word Embeddings for Multilingual Sentiment Analysis
• On the Accuracy of CRNNs for Line-Based OCR: A Multi-Parameter Evaluation
• Per-Pixel Feedback for improving Semantic Segmentation
• A deep neural network for multi-species fish detection using multiple acoustic cameras
• The Power of Communication in a Distributed Multi-Agent System
• Transient Classification in low SNR Gravitational Wave data using Deep Learning
• OpTorch: Optimized deep learning architectures for resource limited environments