Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models

Abstract

Proprietary and closed APIs are becoming increasingly common to process natural language, and are impacting the practical applications of natural language processing, including few-shot classification. Few-shot classification involves training a model to perform a new classification task with a handful of labeled data. This paper presents three contributions. First, we introduce a scenario where the embedding of a pre-trained model is served through a gated API with compute-cost and data-privacy constraints. Second, we propose a transductive inference, a learning paradigm that has been overlooked by the NLP community. Transductive inference, unlike traditional inductive learning, leverages the statistics of unlabeled data. We also introduce a new parameter-free transductive regularizer based on the Fisher-Rao loss, which can be used on top of the gated API embeddings. This method fully utilizes unlabeled data, does not share any label with the third-party API provider and could serve as a baseline for future research. Third, we propose an improved experimental setting and compile a benchmark of eight datasets involving multiclass classification in four different languages, with up to 151 classes. We evaluate our methods using eight backbone models, along with an episodic evaluation over 1,000 episodes, which demonstrate the superiority of transductive inference over the standard inductive setting.

arXiv commentEMNLP 2023

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Scaling Laws for Neural Language Models
• A Simple Neural Attentive Meta-Learner
• ChatGPT: Jack of all trades, master of none
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• Beyond English-Centric Multilingual Machine Translation
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• True Few-Shot Learning with Language Models
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
• OpenPrompt: An Open-source Framework for Prompt-learning
• MetaICL: Learning to Learn In Context
• Laplacian Regularized Few-Shot Learning
• Do We Still Need Clinical Language Models?
• A Constructive Prediction of the Generalization Error Across Scales
• Adversarial Robustness via Fisher-Rao Regularization
• What are the best systems? New perspectives on NLP Benchmarking
• Beam Search with Bidirectional Strategies for Neural Response Generation
• PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models