ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
arXiv:2109.10126
Abstract
Transformer-based language models (LMs) pretrained on large text collections are proven to store a wealth of semantic knowledge. However, 1) they are not effective as sentence encoders when used off-the-shelf, and 2) thus typically lag behind conversationally pretrained (e.g., via response selection) encoders on conversational tasks such as intent detection (ID). In this work, we propose ConvFiT, a simple and efficient two-stage procedure which turns any pretrained LM into a universal conversational encoder (after Stage 1 ConvFiT-ing) and task-specialised sentence encoder (after Stage 2). We demonstrate that 1) full-blown conversational pretraining is not required, and that LMs can be quickly transformed into effective conversational encoders with much smaller amounts of unannotated data; 2) pretrained LMs can be fine-tuned into task-specialised sentence encoders, optimised for the fine-grained semantics of a particular task. Consequently, such specialised sentence encoders allow for treating ID as a simple semantic similarity task based on interpretable nearest neighbours retrieval. We validate the robustness and versatility of the ConvFiT framework with such similarity-based inference on the standard ID evaluation sets: ConvFiT-ed LMs achieve state-of-the-art ID performance across the board, with particular gains in the most challenging, few-shot setups.
arXiv commentEMNLP 2021 (long paper)
References in corpus (10)
- Decoupled Weight Decay Regularization
- Prototypical Networks for Few-shot Learning
- Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
- Training Neural Response Selection for Task-Oriented Dialogue Systems
- Conversational Contextual Cues: The Case of Personalization and History for Response Ranking
- Neural Data Augmentation via Example Extrapolation
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference
- A Neighbourhood Framework for Resource-Lean Content Flagging