LoRA: Low-Rank Adaptation of Large Language Models

Abstract

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

Draft V2 includes better baselines, experiments on GLUE, and more on adapter latency

References in corpus (7)

• Language Models are Few-Shot Learners
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Parameter-Efficient Transfer Learning for NLP
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian

Cited by in corpus (26)

• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• Understanding Large-Language Model (LLM)-powered Human-Robot Interaction
• Large Language Models as Zero-Shot Conversational Recommenders
• LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
• Towards an astronomical foundation model for stars with a Transformer-based model
• Achieving Peak Performance for Large Language Models: A Systematic Review
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Potentials of the Metaverse for Robotized Applications in Industry 4.0 and Industry 5.0
• Entity Recognition from Colloquial Text
• RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolution
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
• DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• TickIt: Leveraging Large Language Models for Automated Ticket Escalation
• Textual interpretation of transient image classifications from large language models
• Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation
• FlanEC: Exploring Flan-T5 for Post-ASR Error Correction
• Enhancing Financial Domain Adaptation of Language Models via Model Augmentation
• Look Ahead Text Understanding and LLM Stitching
• Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A Case Study in Marathi
• StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation