Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers

Abstract

Text classification is a natural language processing (NLP) task relevant to many commercial applications, like e-commerce and customer service. Naturally, classifying such excerpts accurately often represents a challenge, due to intrinsic language aspects, like irony and nuance. To accomplish this task, one must provide a robust numerical representation for documents, a process known as embedding. Embedding represents a key NLP field nowadays, having faced a significant advance in the last decade, especially after the introduction of the word-to-vector concept and the popularization of Deep Learning models for solving NLP tasks, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer-based Language Models (TLMs). Despite the impressive achievements in this field, the literature coverage regarding generating embeddings for Brazilian Portuguese texts is scarce, especially when considering commercial user reviews. Therefore, this work aims to provide a comprehensive experimental study of embedding approaches targeting a binary sentiment classification of user reviews in Brazilian Portuguese. This study includes from classical (Bag-of-Words) to state-of-the-art (Transformer-based) NLP models. The methods are evaluated with five open-source databases with pre-defined data partitions made available in an open digital repository to encourage reproducibility. The Fine-tuned TLMs achieved the best results for all cases, being followed by the Feature-based TLM, LSTM, and CNN, with alternate ranks, depending on the database under analysis.

arXiv comment14 pages, 6 figures. Published at Springer Neural Computing and Applications journal

References in corpus (8)

• Scikit-learn: Machine Learning in Python
• Efficient Estimation of Word Representations in Vector Space
• Text Classification Algorithms: A Survey
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Deep Learning Based Text Classification: A Comprehensive Review
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• A Survey on Text Classification: From Shallow to Deep Learning