Deep Sequential Neural Network

Abstract

Neural Networks sequentially build high-level features through their successive layers. We propose here a new neural network model where each layer is associated with a set of candidate mappings. When an input is processed, at each layer, one mapping among these candidates is selected according to a sequential decision process. The resulting model is structured according to a DAG like architecture, so that a path from the root to a leaf node defines a sequence of transformations. Instead of considering global transformations, like in classical multilayer networks, this model allows us for learning a set of local transformations. It is thus able to process data with different characteristics through specific sequences of such local transformations, increasing the expression power of this model w.r.t a classical multilayered network. The learning algorithm is inspired from policy gradient techniques coming from the reinforcement learning domain and is used here instead of the classical back-propagation based gradient descent techniques. Experiments on different datasets show the relevance of this approach.

Cited by in corpus (9)

• Adaptive Computation Time for Recurrent Neural Networks
• MaskConnect: Connectivity Learning by Gradient Descent
• BlockDrop: Dynamic Inference Paths in Residual Networks
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Computing a human-like reaction time metric from stable recurrent vision models
• URNet : User-Resizable Residual Networks with Conditional Gating Module
• Sequential Coordination of Deep Models for Learning Visual Arithmetic
• Reinforced Decision Trees