Pair-Level Supervised Contrastive Learning for Natural Language Inference

Abstract

Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer the relationship between the sentence pair (premise and hypothesis). Many recent works have used contrastive learning by incorporating the relationship of the sentence pair from NLI datasets to learn sentence representation. However, these methods only focus on comparisons with sentence-level representations. In this paper, we propose a Pair-level Supervised Contrastive Learning approach (PairSCL). We adopt a cross attention module to learn the joint representations of the sentence pairs. A contrastive learning objective is designed to distinguish the varied classes of sentence pairs by pulling those in one class together and pushing apart the pairs in other classes. We evaluate PairSCL on two public datasets of NLI where the accuracy of PairSCL outperforms other methods by 2.1% on average. Furthermore, our method outperforms the previous state-of-the-art method on seven transfer tasks of text classification.

arXiv commentAccepted at ICASSP 2022

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• A Simple Framework for Contrastive Learning of Visual Representations
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Enhanced LSTM for Natural Language Inference
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Natural Language Inference over Interaction Space
• Towards Natural Language Interfaces for Data Visualization: A Survey

Cited by in corpus (1)

• Extracting Sentence Embeddings from Pretrained Transformer Models