Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

Abstract

Aspect based sentiment analysis aims to identify the sentimental tendency towards a given aspect in text. Fine-tuning of pretrained BERT performs excellent on this task and achieves state-of-the-art performances. Existing BERT-based works only utilize the last output layer of BERT and ignore the semantic knowledge in the intermediate layers. This paper explores the potential of utilizing BERT intermediate layers to enhance the performance of fine-tuning of BERT. To the best of our knowledge, no existing work has been done on this research. To show the generality, we also apply this approach to a natural language inference task. Experimental results demonstrate the effectiveness and generality of the proposed approach.

5 pages, 2 figures

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information

Cited by in corpus (5)

• Pre-trained Models for Natural Language Processing: A Survey
• Depression detection in social media posts using transformer-based models and auxiliary features
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
• T-BERT -- Model for Sentiment Analysis of Micro-blogs Integrating Topic Model and BERT
• Investigation of BERT Model on Biomedical Relation Extraction Based on Revised Fine-tuning Mechanism