Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Abstract

Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed ......

25 pages, 12 figures. Due to word limit, the abstract here is truncated. The full abstract is available in the PDF

Cited by in corpus (7)

• Real-Time Decision-Making for Digital Twin in Additive Manufacturing with Model Predictive Control using Time-Series Deep Neural Networks
• Large language models for automated scholarly paper review: A survey
• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations
• Decision-Focused Fine-Tuning of Time Series Foundation Models for Dispatchable Feeder Optimization
• A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications
• MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation
• Artificial intelligence for representing and characterizing quantum systems