Knowledge Efficient Deep Learning for Natural Language Processing

Abstract

Deep learning has become the workhorse for a wide range of natural language processing applications. But much of the success of deep learning relies on annotated examples. Annotation is time-consuming and expensive to produce at scale. Here we are interested in methods for reducing the required quantity of annotated data -- by making the learning methods more knowledge efficient so as to make them more applicable in low annotation (low resource) settings. There are various classical approaches to making the models more knowledge efficient such as multi-task learning, transfer learning, weakly supervised and unsupervised learning etc. This thesis focuses on adapting such classical methods to modern deep learning models and algorithms. This thesis describes four works aimed at making machine learning models more knowledge efficient. First, we propose a knowledge rich deep learning model (KRDL) as a unifying learning framework for incorporating prior knowledge into deep models. In particular, we apply KRDL built on Markov logic networks to denoise weak supervision. Second, we apply a KRDL model to assist the machine reading models to find the correct evidence sentences that can support their decision. Third, we investigate the knowledge transfer techniques in multilingual setting, where we proposed a method that can improve pre-trained multilingual BERT based on the bilingual dictionary. Fourth, we present an episodic memory network for language modelling, in which we encode the large external knowledge for the pre-trained GPT.

arXiv commentPh.D thesis

References in corpus (31)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Prototypical Networks for Few-shot Learning
• Teaching Machines to Read and Comprehend
• Semi-Supervised Learning with Deep Generative Models
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Phrase-Based & Neural Unsupervised Machine Translation
• Character-based Neural Machine Translation
• Contextual LSTM (CLSTM) models for Large scale NLP tasks
• Quasar: Datasets for Question Answering by Search and Reading
• Learning the Structure of Generative Models without Labeled Data
• Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
• Large Memory Layers with Product Keys
• Learning to Compute Word Embeddings On the Fly
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
• Hierarchical Memory Networks
• Memory Augmented Neural Networks with Wormhole Connections
• PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
• Linguistic Knowledge as Memory for Recurrent Neural Networks
• A Co-Matching Model for Multi-choice Reading Comprehension
• Dynamic Fusion Networks for Machine Reading Comprehension
• Visual Question Answering with Memory-Augmented Networks
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse