Achieving Peak Performance for Large Language Models: A Systematic Review

Abstract

In recent years, large language models (LLMs) have achieved remarkable success in natural language processing (NLP). LLMs require an extreme amount of parameters to attain high performance. As models grow into the trillion-parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models. Optimizing LLM performance involves two main approaches: fine-tuning pre-trained models for specific tasks to achieve state-of-the-art performance, and reducing costs or improving training time while maintaining similar performance. This paper presents a systematic literature review (SLR) following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We reviewed 65 publications out of 983 from 2017 to December 2023, retrieved from 5 databases. The study presents methods to optimize and accelerate LLMs while achieving cutting-edge results without sacrificing accuracy. We begin with an overview of the development of language modeling, followed by a detailed explanation of commonly used frameworks and libraries, and a taxonomy for improving and speeding up LLMs based on three classes: LLM training, LLM inference, and system serving. We then delve into recent optimization and acceleration strategies such as training optimization, hardware optimization, scalability and reliability, accompanied by the taxonomy and categorization of these strategies. Finally, we provide an in-depth comparison of each class and strategy, with two case studies on optimizing model training and enhancing inference efficiency. These case studies showcase practical approaches to address LLM resource limitations while maintaining performance.

34 pages, 7 figures, 8 tables. Journal Article: IEEE Access

References in corpus (15)

• LLaMA: Open and Efficient Foundation Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Scaling Laws for Neural Language Models
• Galactica: A Large Language Model for Science
• Instruction Tuning with GPT-4
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Accelerating LLM Inference with Staged Speculative Decoding
• Fast Inference of Mixture-of-Experts Language Models with Offloading
• Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
• GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length
• Multilevel Large Language Models for Everyone
• FPTQ: Fine-grained Post-Training Quantization for Large Language Models
• FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
• AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models