Large Language Models: A Survey

Abstract

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Cited by in corpus (11)

• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• The Dawn of Natural Language to SQL: Are We Fully Ready?
• A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods
• GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks
• Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges
• Enhancing Trust and Safety in Digital Payments: An LLM-Powered Approach
• AdditiveLLM: Large Language Models Predict Defects in Additive Manufacturing
• On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing
• Large Language Models -- the Future of Fundamental Physics?
• A Roadmap for Tamed Interactions with Large Language Models
• Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis