Improving the Expressiveness of Deep Learning Frameworks with Recursion

Abstract

Recursive neural networks have widely been used by researchers to handle applications with recursively or hierarchically structured data. However, embedded control flow deep learning frameworks such as TensorFlow, Theano, Caffe2, and MXNet fail to efficiently represent and execute such neural networks, due to lack of support for recursion. In this paper, we add recursion to the programming model of existing frameworks by complementing their design with recursive execution of dataflow graphs as well as additional APIs for recursive definitions. Unlike iterative implementations, which can only understand the topological index of each node in recursive data structures, our recursive implementation is able to exploit the recursive relationships between nodes for efficient execution based on parallel computation. We present an implementation on TensorFlow and evaluation results with various recursive neural network models, showing that our recursive implementation not only conveys the recursive nature of recursive neural networks better than other implementations, but also uses given resources more effectively to reduce training and inference time.

arXiv commentAppeared in EuroSys 2018. 13 pages, 11 figures

References in corpus (6)

• The NumPy array: a structure for efficient numerical computation
• TensorFlow: A system for large-scale machine learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• DyNet: The Dynamic Neural Network Toolkit
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Deep Learning with Dynamic Computation Graphs

Cited by in corpus (6)

• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Cortex: A Compiler for Recursive Deep Learning Models
• SoD${}^2$: Statically Optimizing Dynamic Deep Neural Network
• Automatic Differentiation for ML-family languages: correctness via logical relations
• No DNN Left Behind: Improving Inference in the Cloud with Multi-Tenancy