LoRA: Low-Rank Adaptation of Large Language Models

Abstract

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

arXiv commentDraft V2 includes better baselines, experiments on GLUE, and more on adapter latency

References in corpus (18)

• Decoupled Weight Decay Regularization
• Language Models are Few-Shot Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Predicting Parameters in Deep Learning
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning multiple visual domains with residual adapters
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Parameter-Efficient Transfer Learning for NLP
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• When Do Neural Networks Outperform Kernel Methods?
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian

Cited by in corpus (144)

• On the Opportunities and Risks of Foundation Models
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
• Recommender Systems in the Era of Large Language Models (LLMs)
• Large language models in medicine: the potentials and pitfalls
• Towards a Unified View of Parameter-Efficient Transfer Learning
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• Understanding Large-Language Model (LLM)-powered Human-Robot Interaction
• Large Language Models as Zero-Shot Conversational Recommenders
• Break-A-Scene: Extracting Multiple Concepts from a Single Image
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
• Materials science in the era of large language models: a perspective
• Diffusion Model-Based Image Editing: A Survey
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• From task structures to world models: What do LLMs know?
• Learning to Decode the Surface Code with a Recurrent, Transformer-Based Neural Network
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Large Language Models Can Be Strong Differentially Private Learners
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• Tracking and Mapping in Medical Computer Vision: A Review
• Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness
• DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
• Differentially Private Fine-tuning of Language Models
• Fine-tuning and Utilization Methods of Domain-specific LLMs
• Mask-guided BERT for Few Shot Text Classification
• Towards an astronomical foundation model for stars with a Transformer-based model
• Enhanced Automated Code Vulnerability Repair using Large Language Models
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• Enhancing textual textbook question answering with large language models and retrieval augmented generation
• Quo Vadis ChatGPT? From Large Language Models to Large Knowledge Models
• Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights
• Towards Continual Knowledge Learning of Language Models
• Achieving Peak Performance for Large Language Models: A Systematic Review
• Mobile Foundation Model as Firmware
• GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
• Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
• Domain-specific ChatBots for Science using Embeddings
• PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models
• Cross-Modal Adapter for Vision-Language Retrieval
• The Chosen One: Consistent Characters in Text-to-Image Diffusion Models
• Embedded Visual Prompt Tuning
• Exploring Gen-AI applications in building research and industry: A review
• OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Design
• Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach
• A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• Opportunities and Challenges of Generative-AI in Finance
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• Llama-Mob: Instruction-Tuning Llama-3-8B Excels in City-Scale Mobility Prediction
• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations
• A Few-Shot Approach to Dysarthric Speech Intelligibility Level Classification Using Transformers
• Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly
• EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
• Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning
• ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation
• A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model
• Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts
• Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge
• Kvasir-VQA: A Text-Image Pair GI Tract Dataset
• LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot
• Potentials of the Metaverse for Robotized Applications in Industry 4.0 and Industry 5.0
• Entity Recognition from Colloquial Text
• Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
• CANAL -- Cyber Activity News Alerting Language Model: Empirical Approach vs. Expensive LLM
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolution
• VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities
• Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation
• A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
• Large Language Models for Combinatorial Optimization: A Systematic Review
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
• LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration
• Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification
• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• ForgetMe: Evaluating Selective Forgetting in Generative Models
• Precise Length Control in Large Language Models
• Hyperspectral Dataset and Deep Learning methods for Waste from Electric and Electronic Equipment Identification (WEEE)
• On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software
• Complex QA and language models hybrid architectures, Survey
• A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications
• ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answering
• DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
• Research on Tibetan Tourism Viewpoints information generation system based on LLM
• Compressing Neural Networks Using Tensor Networks with Exponentially Fewer Variational Parameters
• A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
• Exploring the Impact of Model Scaling on Parameter-Efficient Tuning
• TickIt: Leveraging Large Language Models for Automated Ticket Escalation
• FANAL -- Financial Activity News Alerting Language Modeling Framework
• Automated Description Generation for Software Patches
• EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering
• AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
• Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
• Textual interpretation of transient image classifications from large language models
• Enhancing Financial Domain Adaptation of Language Models via Model Augmentation
• FlanEC: Exploring Flan-T5 for Post-ASR Error Correction
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints
• Multi-Head Adapter Routing for Cross-Task Generalization
• Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation
• Analysis of the Two-Step Heterogeneous Transfer Learning for Laryngeal Blood Vessel Classification: Issue and Improvement
• From Pampas to Pixels: Fine-Tuning Diffusion Models for Gaúcho Heritage
• GEMRec: Towards Generative Model Recommendation
• Leveraging Label Semantics and Meta-Label Refinement for Multi-Label Question Classification
• SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors
• Pie: A Programmable Serving System for Emerging LLM Applications
• Look Ahead Text Understanding and LLM Stitching
• AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
• The Complexity Dynamics of Grokking
• MAX: Masked Autoencoder for X-ray Fluorescence in Geological Investigation
• A Hybrid Architecture with Efficient Fine Tuning for Abstractive Patent Document Summarization
• Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
• Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A Case Study in Marathi
• Generating clickbait spoilers with an ensemble of large language models
• Data-driven Prediction of Ionic Conductivity in Solid-State Electrolytes with Machine Learning and Large Language Models
• Event-Free Moving Object Segmentation from Moving Ego Vehicle
• Efficient Extraction of Noise-Robust Discrete Units from Self-Supervised Speech Models
• A Unified Approach to Emotion Detection and Task-Oriented Dialogue Modeling
• Understanding Personalization Gains in Whisper ASR
• The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning
• Privacy-Aware Document Visual Question Answering
• StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation
• Large Language Models -- the Future of Fundamental Physics?
• mmPencil: Toward Writing-Style-Independent In-Air Handwriting Recognition via mmWave Radar and Large Vision-Language Model
• SOP^2: Transfer Learning with Scene-Oriented Prompt Pool on 3D Object Detection
• Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation
• ThRIve: Thermally Robust CNN Inference via Low-Rank Adaptation in Heterogeneous PIM Architectures
• Tensorization of neural networks for improved privacy and interpretability
• Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts