OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction
arXiv:2105.05535
Abstract
We propose an ensemble model for predicting the lexical complexity of words and multiword expressions (MWEs). The model receives as input a sentence with a target word or MWEand outputs its complexity score. Given that a key challenge with this task is the limited size of annotated data, our model relies on pretrained contextual representations from different state-of-the-art transformer-based language models (i.e., BERT and RoBERTa), and on a variety of training methods for further enhancing model generalization and robustness:multi-step fine-tuning and multi-task learning, and adversarial training. Additionally, we propose to enrich contextual representations by adding hand-crafted features during training. Our model achieved competitive results and ranked among the top-10 systems in both sub-tasks.
References in corpus (11)
- Natural Language Processing (almost) from Scratch
- An Overview of Multi-Task Learning in Deep Neural Networks
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
- A Surprisingly Robust Trick for Winograd Schema Challenge
- Adversarial Training for Large Neural Language Models
- Robust Neural Machine Translation with Doubly Adversarial Inputs
- CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data
- The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
- Complex Word Identification: Challenges in Data Annotation and System Performance
- Adversarial Training for Commonsense Inference