Applying SoftTriple Loss for Supervised Language Model Fine Tuning

Abstract

We introduce a new loss function TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models based on cross-entropy and SoftTriple loss. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samples in the training dataset, the higher gain -- thus, for small-sized dataset it is 0.78%, for medium-sized -- 0.86% for large -- 0.20% and for extra-large 0.04%.

References in corpus (7)

• A Simple Framework for Contrastive Learning of Visual Representations
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• In Defense of the Triplet Loss for Person Re-Identification
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Metric learning by Similarity Network for Deep Semi-Supervised Learning

Cited by in corpus (1)

• Few-shot learning for automated content analysis: Efficient coding of arguments and claims in the debate on arms deliveries to Ukraine