Semantics, Representations and Grammars for Deep Learning

Abstract

Deep learning is currently the subject of intensive study. However, fundamental concepts such as representations are not formally defined -- researchers "know them when they see them" -- and there is no common language for describing and analyzing algorithms. This essay proposes an abstract framework that identifies the essential features of current practice and may provide a foundation for future developments. The backbone of almost all deep learning algorithms is backpropagation, which is simply a gradient computation distributed over a neural network. The main ingredients of the framework are thus, unsurprisingly: (i) game theory, to formalize distributed optimization; and (ii) communication protocols, to track the flow of zeroth and first-order information. The framework allows natural definitions of semantics (as the meaning encoded in functions), representations (as functions whose semantics is chosen to optimized a criterion) and grammars (as communication protocols equipped with first-order convergence guarantees). Much of the essay is spent discussing examples taken from the literature. The ultimate aim is to develop a graphical language for describing the structure of deep learning algorithms that backgrounds the details of the optimization procedure and foregrounds how the components interact. Inspiration is taken from probabilistic graphical models and factor graphs, which capture the essential structural features of multivariate distributions.

20 pages, many diagrams

References in corpus (19)

• Deep Learning in Neural Networks: An Overview
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Continuous control with deep reinforcement learning
• Theano: new features and speed improvements
• Train faster, generalize better: Stability of stochastic gradient descent
• Gradient Estimation Using Stochastic Computation Graphs
• Blocks and Fuel: Frameworks for deep learning
• Fast Convergence of Regularized Learning in Games
• Random feedback weights support learning in deep neural networks
• The operad of wiring diagrams: formalizing a graphical language for databases, recursion, and plug-and-play circuits
• A Collaborative Mechanism for Crowdsourcing Prediction Problems
• Machine Learning Markets
• Automatic Differentiation of Algorithms for Machine Learning
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
• On Lower and Upper Bounds for Smooth and Strongly Convex Optimization Problems
• Randomized co-training: from cortical neurons to machine learning and back again
• Implicit Temporal Differences
• Deep Online Convex Optimization by Putting Forecaster to Sleep