A Survey on In-context Learning

Abstract

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Update

Cited by in corpus (17)

• Recommender Systems in the Era of Large Language Models (LLMs)
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• ChatEDA: A Large Language Model Powered Autonomous Agent for EDA
• Harms from Increasingly Agentic Algorithmic Systems
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels
• "It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science
• "I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation
• Potential Benefits of Employing Large Language Models in Research in Moral Education and Development
• From system models to class models: An in-context learning paradigm
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness
• Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?
• WiseMind: a knowledge-guided multi-agent framework for accurate and empathetic psychiatric diagnosis
• Incorporating Large Language Models into Production Systems for Enhanced Task Automation and Flexibility