Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

Abstract

Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

Accepted by EMNLP 2023 Findings

References in corpus (11)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• On Calibration of Modern Neural Networks
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT
• Generate rather than Retrieve: Large Language Models are Strong Context Generators
• Selective Annotation Makes Language Models Better Few-Shot Learners
• Exploring the Feasibility of ChatGPT for Event Extraction
• InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction
• Recitation-Augmented Language Models
• PIVOINE: Instruction Tuning for Open-world Information Extraction

Cited by in corpus (5)

• Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
• Mining experimental data from Materials Science literature with Large Language Models: an evaluation study
• A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness
• Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities
• Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction