Learning Neural Models for Natural Language Processing in the Face of Distributional Shift

Abstract

The dominating NLP paradigm of training a strong neural predictor to perform one task on a specific dataset has led to state-of-the-art performance in a variety of applications (eg. sentiment classification, span-prediction based question answering or machine translation). However, it builds upon the assumption that the data distribution is stationary, ie. that the data is sampled from a fixed distribution both at training and test time. This way of training is inconsistent with how we as humans are able to learn from and operate within a constantly changing stream of information. Moreover, it is ill-adapted to real-world use cases where the data distribution is expected to shift over the course of a model's lifetime. The first goal of this thesis is to characterize the different forms this shift can take in the context of natural language processing, and propose benchmarks and evaluation metrics to measure its effect on current deep learning architectures. We then proceed to take steps to mitigate the effect of distributional shift on NLP models. To this end, we develop methods based on parametric reformulations of the distributionally robust optimization framework. Empirically, we demonstrate that these approaches yield more robust models as demonstrated on a selection of realistic problems. In the third and final part of this thesis, we explore ways of efficiently adapting existing models to new domains or tasks. Our contribution to this topic takes inspiration from information geometry to derive a new gradient update rule which alleviate catastrophic forgetting issues during adaptation.

arXiv commentPhD thesis

References in corpus (38)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Overcoming catastrophic forgetting in neural networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• End to End Learning for Self-Driving Cars
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Rényi Divergence and Kullback-Leibler Divergence
• Pixel Recurrent Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Classifier Technology and the Illusion of Progress
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• The Microsoft 2017 Conversational Speech Recognition System
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• DyNet: The Dynamic Neural Network Toolkit
• On Tiny Episodic Memories in Continual Learning
• Adversarial Perturbations Against Deep Neural Networks for Malware Classification
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Training Tips for the Transformer Model
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• Towards Robust Evaluations of Continual Learning
• On Adversarial Examples for Character-Level Neural Machine Translation
• Towards Crafting Text Adversarial Samples
• Diachronic word embeddings and semantic shifts: a survey
• Deceiving Google's Perspective API Built for Detecting Toxic Comments
• Uncertainty-based Continual Learning with Adaptive Regularization
• Uniform convergence may be unable to explain generalization in deep learning
• Does Distributionally Robust Supervised Learning Give Robust Classifiers?
• Training verified learners with learned verifiers
• XNMT: The eXtensible Neural Machine Translation Toolkit
• Learning with Average Top-k Loss
• Robustness Verification for Transformers
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
• Robust Bayesian Classification Using an Optimistic Score Ratio
• Modeling the Second Player in Distributionally Robust Optimization