Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction

Abstract

Our entry into the HAHA 2019 Challenge placed $3^{rd}$ in the classification task and $2^{nd}$ in the regression task. We describe our system and innovations, as well as comparing our results to a Naive Bayes baseline. A large Twitter based corpus allowed us to train a language model from scratch focused on Spanish and transfer that knowledge to our competition model. To overcome the inherent errors in some labels we reduce our class confidence with label smoothing in the loss function. All the code for our project is included in a GitHub repository for easy reference and to enable replication by others.

arXiv commentIberLEF 2019 Workshop

References in corpus (6)

• SMOTE: Synthetic Minority Over-sampling Technique
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Regularizing and Optimizing LSTM Language Models
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Quasi-Recurrent Neural Networks
• Universal Language Model Fine-Tuning with Subword Tokenization for Polish

Cited by in corpus (1)

• Integrating extracted information from bert and multiple embedding methods with the deep neural network for humour detection