Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
arXiv:2311.06102 · doi:10.1145/3604237.3626891
Abstract
Standard Full-Data classifiers in NLP demand thousands of labeled examples, which is impractical in data-limited domains. Few-shot methods offer an alternative, utilizing contrastive learning techniques that can be effective with as little as 20 examples per class. Similarly, Large Language Models (LLMs) like GPT-4 can perform effectively with just 1-5 examples per class. However, the performance-cost trade-offs of these methods remain underexplored, a critical concern for budget-limited organizations. Our work addresses this gap by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation of cutting-edge LLMs by OpenAI, Cohere, and Anthropic in a comprehensive set of few-shot scenarios. We complete the picture with two additional methods: first, a cost-effective querying method for LLMs based on retrieval-augmented generation (RAG), able to reduce operational costs multiple times compared to classic few-shot approaches, and second, a data augmentation method using GPT-4, able to improve performance in data-limited scenarios. Finally, to inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis.
arXiv commentLong paper accepted to ACM ICAIF-23
References in corpus (11)
- A Simple Framework for Contrastive Learning of Visual Representations
- Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Knowledge Distillation: A Survey
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- MPNet: Masked and Permuted Pre-training for Language Understanding
- Efficient Few-Shot Learning Without Prompts
- FiNER: Financial Numeric Entity Recognition for XBRL Tagging
- Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance
- Financial misstatement detection: a realistic evaluation