Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks

Abstract

Recent deep learning (DL) models have moved beyond static network architectures to dynamic ones, handling data where the network structure changes every example, such as sequences of variable lengths, trees, and graphs. Existing dataflow-based programming models for DL---both static and dynamic declaration---either cannot readily express these dynamic models, or are inefficient due to repeated dataflow graph construction and processing, and difficulties in batched execution. We present Cavs, a vertex-centric programming interface and optimized system implementation for dynamic DL models. Cavs represents dynamic network structure as a static vertex function $\mathcal{F}$ and a dynamic instance-specific graph $\mathcal{G}$, and performs backpropagation by scheduling the execution of $\mathcal{F}$ following the dependencies in $\mathcal{G}$. Cavs bypasses expensive graph construction and preprocessing overhead, allows for the use of static graph optimization techniques on pre-defined operations in $\mathcal{F}$, and naturally exposes batched execution opportunities over different graphs. Experiments comparing Cavs to two state-of-the-art frameworks for dynamic NNs (TensorFlow Fold and DyNet) demonstrate the efficacy of this approach: Cavs achieves a near one order of magnitude speedup on training of various dynamic NN architectures, and ablations demonstrate the contribution of our proposed batching and memory management strategies.

Short versions of this paper were presented at AISys workshop@SOSP 2017 and MLSys workshop@NIPS 2017

References in corpus (17)

• The NumPy array: a structure for efficient numerical computation
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: A system for large-scale machine learning
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Recurrent Neural Network Regularization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• DyNet: The Dynamic Neural Network Toolkit
• Efficient softmax approximation for GPUs
• Deep Learning with Dynamic Computation Graphs
• Combining the Best of Convolutional Layers and Recurrent Layers: A Hybrid Network for Semantic Segmentation
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• On-the-fly Operation Batching in Dynamic Computation Graphs
• Approximation-Aware Dependency Parsing by Belief Propagation

Cited by in corpus (1)

• Just-in-Time Dynamic-Batching