Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
arXiv:2009.07411
Abstract
Syntax has been shown useful for various NLP tasks, while existing work mostly encodes singleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledge Distillation, to integrate heterogeneous structure knowledge into a unified sequential LSTM encoder. Experimental results on four typical syntax-dependent tasks show that our method outperforms tree encoders by effectively integrating rich heterogeneous structure syntax, meanwhile reducing error propagation, and also outperforms ensemble methods, in terms of both the efficiency and accuracy.
arXiv commentTo appear at EMNLP2020
References in corpus (17)
- Distilling the Knowledge in a Neural Network
- Enhanced LSTM for Natural Language Inference
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- A large annotated corpus for learning natural language inference
- Born Again Neural Networks
- Deep Learning with Dynamic Computation Graphs
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
- Three Generative, Lexicalised Models for Statistical Parsing
- Constituency Parsing with a Self-Attentive Encoder
- Technical report on Conversational Question Answering
- Sentence-State LSTM for Text Representation
- Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
- Head-Driven Phrase Structure Grammar Parsing on Penn Treebank
- Sequence Labeling Parsing by Learning Across Representations
- A* CCG Parsing with a Supertag and Dependency Factored Model
- Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing
- Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation