HUBERT Untangles BERT to Improve Transfer across NLP Tasks
arXiv:1910.12647
Abstract
We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- A large annotated corpus for learning natural language inference
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Visualizing and Measuring the Geometry of BERT
- On orthogonality and learning recurrent networks with long term dependencies
- BERT Rediscovers the Classical NLP Pipeline
- Unifying Question Answering, Text Classification, and Regression via Span Extraction
- Learning to Reason with Third-Order Tensor Products
- Open Sesame: Getting Inside BERT's Linguistic Knowledge
- Question-Answering with Grammatically-Interpretable Representations
Cited by in corpus (5)
- Syntactic Data Augmentation Increases Robustness to Inference Heuristics
- GAN Memory with No Forgetting
- Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
- From `Snippet-lects' to Doculects and Dialects: Leveraging Neural Representations of Speech for Placing Audio Signals in a Language Landscape
- Disentangling Representations of Text by Masking Transformers