Regularizing Recurrent Neural Networks via Sequence Mixup

Abstract

In this paper, we extend a class of celebrated regularization techniques originally proposed for feed-forward neural networks, namely Input Mixup (Zhang et al., 2017) and Manifold Mixup (Verma et al., 2018), to the realm of Recurrent Neural Networks (RNN). Our proposed methods are easy to implement and have a low computational complexity, while leverage the performance of simple neural architectures in a variety of tasks. We have validated our claims through several experiments on real-world datasets, and also provide an asymptotic theoretical analysis to further investigate the properties and potential impacts of our proposed techniques. Applying sequence mixup to BiLSTM-CRF model (Huang et al., 2015) to Named Entity Recognition task on CoNLL-2003 data (Sang and De Meulder, 2003) has improved the F-1 score on the test stage and reduced the loss, considerably.

arXiv comment17 pages

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• On the difficulty of training Recurrent Neural Networks
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Recurrent Neural Network Regularization
• Regularizing and Optimizing LSTM Language Models
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations