Deep Learning with Dynamic Computation Graphs

Abstract

Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.

Published as a conference paper at ICLR 2017

References in corpus (2)

• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Learning to Compose Neural Networks for Question Answering

Cited by in corpus (33)

• Learning to Generate Reviews and Discovering Sentiment
• DyNet: The Dynamic Neural Network Toolkit
• Ray: A Distributed Framework for Emerging AI Applications
• Fashionable Modelling with Flux
• Relay: A New IR for Machine Learning Frameworks
• Adaptive Graph Convolutional Neural Networks
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• GamePad: A Learning Environment for Theorem Proving
• Deep Learning-based Sentiment Classification: A Comparative Survey
• Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs
• Adorym: A multi-platform generic x-ray image reconstruction framework based on automatic differentiation
• AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
• On-the-fly Operation Batching in Dynamic Computation Graphs
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Combining Convolution and Recursive Neural Networks for Sentiment Analysis
• Certified Data Removal from Machine Learning Models
• Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator
• Cortex: A Compiler for Recursive Deep Learning Models
• Improving the Expressiveness of Deep Learning Frameworks with Recursion
• Learning Graph While Training: An Evolving Graph Convolutional Neural Network
• Dynamic-structured Semantic Propagation Network
• Auto-Vectorizing TensorFlow Graphs: Jacobians, Auto-Batching And Beyond
• Cell-aware Stacked LSTMs for Modeling Sentences
• An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms
• Cooperative Learning of Disjoint Syntax and Semantics
• Recursive Variational Autoencoders for 3D Blood Vessel Generative Modeling
• Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
• Automatically Batching Control-Intensive Programs for Modern Accelerators
• A Survey on Semantic Parsing from the perspective of Compositionality
• Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks
• Effective Approaches to Batch Parallelization for Dynamic Neural Network Architectures
• Vampire With a Brain Is a Good ITP Hammer
• Just-in-Time Dynamic-Batching