StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
arXiv:2012.00857
Abstract
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words. While previous unsupervised parsing methods mostly focus on only inducing one class of grammars, we introduce a novel model, StructFormer, that can simultaneously induce dependency and constituency structure. To achieve this, we propose a new parsing framework that can jointly generate a constituency tree and dependency graph. Then we integrate the induced dependency relations into the transformer, in a differentiable manner, through a novel dependency-constrained self-attention mechanism. Experimental results show that our model can achieve strong results on unsupervised constituency parsing, unsupervised dependency parsing, and masked language modeling at the same time.
arXiv commentPublished as a conference paper at ACL 2021
References in corpus (8)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
- A Systematic Assessment of Syntactic Generalization in Neural Language Models
- Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
- Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
- Unsupervised Dependency Parsing: Let's Use Supervised Parsers
- CRF Autoencoder for Unsupervised Dependency Parsing
- Dependency Grammar Induction with Neural Lexicalization and Big Training Data