Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Abstract

Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.

11 pages

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Skip-Thought Vectors
• A C-LSTM Neural Network for Text Classification
• Discriminative Neural Sentence Modeling by Tree-Based Convolution
• Learning text representation using recurrent convolutional neural network with highway layers

Cited by in corpus (36)

• Deep Learning Based Text Classification: A Comprehensive Review
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Cyberbullying Detection for Low-resource Languages and Dialects: Review of the State of the Art
• A Survey on Text Classification: From Shallow to Deep Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
• Technical Q&A Site Answer Recommendation via Question Boosting
• XAI in Computational Linguistics: Understanding Political Leanings in the Slovenian Parliament
• Language-Independent Sentiment Labelling with Distant Supervision: A Case Study for English, Sepedi and Setswana
• Vulgar Remarks Detection in Chittagonian Dialect of Bangla
• Detection of depression on social networks using transformers and ensembles
• A Data-driven Neural Network Architecture for Sentiment Analysis
• Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers
• When Graph Convolution Meets Double Attention: Online Privacy Disclosure Detection with Multi-Label Text Classification
• Text as Environment: A Deep Reinforcement Learning Text Readability Assessment Model
• Large-Scale News Classification using BERT Language Model: Spark NLP Approach
• Generating Responses Expressing Emotion in an Open-domain Dialogue System
• Densely Connected Bidirectional LSTM with Applications to Sentence Classification
• #phramacovigilance - Exploring Deep Learning Techniques for Identifying Mentions of Medication Intake from Twitter
• Detecting Radical Text over Online Media using Deep Learning
• Sentiment and Emotion Classification of Epidemic Related Bilingual data from Social Media
• Particle Filter Recurrent Neural Networks
• Log Message Anomaly Detection and Classification Using Auto-B/LSTM and Auto-GRU
• Learning Robust, Transferable Sentence Representations for Text Classification
• Development of deep learning algorithms to categorize free-text notes pertaining to diabetes: convolution neural networks achieve higher accuracy than support vector machines
• Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation
• DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder
• SentiMATE: Learning to play Chess through Natural Language Processing
• Adaptive Region Embedding for Text Classification
• Human Activity Recognition using Deep Learning Models on Smartphones and Smartwatches Sensor Data
• Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning
• Relation Detection for Indonesian Language using Deep Neural Network -- Support Vector Machine
• Categorical Metadata Representation for Customized Text Classification
• Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition
• Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks
• Joint Embedding of Words and Category Labels for Hierarchical Multi-label Text Classification