On the Origin of Deep Learning

Abstract

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

70 pages, 200 references

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Understanding deep learning requires rethinking generalization
• Energy-based Generative Adversarial Network
• Deep Convolutional Inverse Graphics Network
• Multiple Object Recognition with Visual Attention
• Residual Networks of Residual Networks: Multilevel Residual Networks
• On the saddle point problem for non-convex optimization
• Calibrating Energy-based Generative Adversarial Networks
• Dropout with Expectation-linear Regularization

Cited by in corpus (5)

• Predict Forex Trend via Convolutional Neural Networks
• Evolving Neural Networks in Reinforcement Learning by means of UMDAc
• Deep Learning Paradigm with Transformed Monolingual Word Embeddings for Multilingual Sentiment Analysis
• Deep Octonion Networks
• Per-Pixel Feedback for improving Semantic Segmentation