DyNet: The Dynamic Neural Network Toolkit

Abstract

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at http://github.com/clab/dynet.

33 pages

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Character-based Neural Machine Translation
• Learning to Compose Neural Networks for Question Answering
• Segmental Recurrent Neural Networks
• Deep Learning with Dynamic Computation Graphs
• Classes for Fast Maximum Entropy Training
• Semi Supervised Preposition-Sense Disambiguation using Multilingual Data

Cited by in corpus (95)

• Improving Text-to-SQL Evaluation Methodology
• Fashionable Modelling with Flux
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Relay: A New IR for Machine Learning Frameworks
• A Transition-Based Directed Acyclic Graph Parser for UCCA
• Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
• Neural Models for Information Retrieval
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Multilingual Neural Machine Translation with Task-Specific Attention
• SparseMAP: Differentiable Sparse Structured Inference
• End-to-end neural relation extraction using deep biaffine attention
• Don't Unroll Adjoint: Differentiating SSA-Form Programs
• XNMT: The eXtensible Neural Machine Translation Toolkit
• TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning
• Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment
• Abstract Syntax Networks for Code Generation and Semantic Parsing
• Learning to Generate Corrective Patches using Neural Machine Translation
• Learning to Select, Track, and Generate for Data-to-Text
• Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
• AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
• On-the-fly Operation Batching in Dynamic Computation Graphs
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
• Attention-based Encoder-Decoder Networks for Spelling and Grammatical Error Correction
• Beyond Graph Neural Networks with Lifted Relational Neural Networks
• Polyglot Semantic Parsing in APIs
• An improved neural network model for joint POS tagging and dependency parsing
• Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning
• Neural Discourse Structure for Text Categorization
• A Minimal Span-Based Neural Constituency Parser
• Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource
• Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator
• Dynamic Entity Representations in Neural Language Models
• Multitask Learning with CTC and Segmental CRF for Speech Recognition
• Neural Lattice-to-Sequence Models for Uncertain Inputs
• A* CCG Parsing with a Supertag and Dependency Factored Model
• Parsing with Traces: An $O(n^4)$ Algorithm and a Structural Representation
• Cortex: A Compiler for Recursive Deep Learning Models
• Improving the Expressiveness of Deep Learning Frameworks with Recursion
• PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
• Disentangling Adaptive Gradient Methods from Learning Rates
• Deep Probabilistic Programming Languages: A Qualitative Study
• LP-SparseMAP: Differentiable Relaxed Optimization for Sparse Structured Prediction
• Towards Neural Decompilation
• To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing
• LazyTensor: combining eager execution with domain-specific compilers
• Learning Joint Semantic Parsers from Disjoint Data
• Model Transfer for Tagging Low-resource Languages using a Bilingual Dictionary
• Two Local Models for Neural Constituent Parsing
• Modelling Verbal Morphology in Nen
• Paraphrase to Explicate: Revealing Implicit Noun-Compound Relations
• EagerPy: Writing Code That Works Natively with PyTorch, TensorFlow, JAX, and NumPy
• Unifying Instance and Panoptic Segmentation with Dynamic Rank-1 Convolutions
• Cross-Domain Generalization of Neural Constituency Parsers
• Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing
• Fast and Accurate Neural Word Segmentation for Chinese
• Context Models for OOV Word Translation in Low-Resource Languages
• Joint Learning of Domain Classification and Out-of-Domain Detection with Dynamic Class Weighting for Satisficing False Acceptance Rates
• Neural Machine Translation via Binary Code Prediction
• A Resource for Studying Chatino Verbal Morphology
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Swift for TensorFlow: A portable, flexible platform for deep learning
• A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding
• Pointer-based Fusion of Bilingual Lexicons into Neural Machine Translation
• Syntactically Look-Ahead Attention Network for Sentence Compression
• Lessons Learned from Applying off-the-shelf BERT: There is no Silver Bullet
• Predicting Semantic Relations using Global Graph Properties
• Low-Resource Syntactic Transfer with Unsupervised Source Reordering
• Exploring Contextualized Neural Language Models for Temporal Dependency Parsing
• Global Transition-based Non-projective Dependency Parsing
• Zero-shot Neural Transfer for Cross-lingual Entity Linking
• Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
• Privacy-preserving Neural Representations of Text
• Cross-Lingual Transfer of Semantic Roles: From Raw Text to Semantic Roles
• A neural joint model for Vietnamese word segmentation, POS tagging and dependency parsing
• Parallelizable Stack Long Short-Term Memory
• A Hybrid Residual Dilated LSTM end Exponential Smoothing Model for Mid-Term Electric Load Forecasting
• What's Going On in Neural Constituency Parsers? An Analysis
• Mutlitask Learning for Cross-Lingual Transfer of Semantic Dependencies
• GPU-Accelerated Primal Learning for Extremely Fast Large-Scale Classification
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks
• OneNet: Joint Domain, Intent, Slot Prediction for Spoken Language Understanding
• Joint Parsing and Generation for Abstractive Summarization
• TensorX: Extensible API for Neural Network Model Design and Deployment
• Speaker-Sensitive Dual Memory Networks for Multi-Turn Slot Tagging
• Dynamic Data Selection for Curriculum Learning via Ability Estimation
• Workload-aware Automatic Parallelization for Multi-GPU DNN Training
• Towards Dynamic Computation Graphs via Sparse Latent Structure
• Supervised Domain Enablement Attention for Personalized Domain Classification
• PrototypeML: A Neural Network Integrated Design and Development Environment
• Transfer Learning Between Related Tasks Using Expected Label Proportions
• Exploring Recombination for Efficient Decoding of Neural Machine Translation
• Augmenting a BiLSTM tagger with a Morphological Lexicon and a Lexical Category Identification Step
• An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation