Guideline Learning for In-context Information Extraction
arXiv:2310.05066 · doi:10.18653/v1/2023.emnlp-main.950
Abstract
Large language models (LLMs) can perform a new task by merely conditioning on task instructions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL). In-context Information Extraction (IE) has recently garnered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art supervised expert models. We highlight a key reason for this shortfall: underspecified task description. The limited-length context struggles to thoroughly express the intricate IE task instructions and various edge cases, leading to misalignment in task comprehension with humans. In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes a set of guidelines based on a few error cases, and during inference, GL retrieves helpful guidelines for better ICL. Moreover, we propose a self-consistency-based active learning method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL can significantly improve the performance of in-context IE.
arXiv commentEMNLP 2023 main conference
References in corpus (10)
- LLaMA: Open and Efficient Foundation Language Models
- PaLM: Scaling Language Modeling with Pathways
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
- ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT
- Exploring the Feasibility of ChatGPT for Event Extraction
- An Empirical Study on Information Extraction using Large Language Models
- GPT-RE: In-context Learning for Relation Extraction using Large Language Models
- Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners
- Learning In-context Learning for Named Entity Recognition