The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints
arXiv:2407.19299 · doi:10.1109/ACCESS.2025.3582037
Abstract
Fine-tuning Large Language Models (LLMs) for clinical Natural Language Processing (NLP) poses significant challenges due to domain gap, limited data, and stringent hardware constraints. In this study, we evaluate four adapter techniques-Adapter, Lightweight, TinyAttention, and Gated Residual Network (GRN) - equivalent to Low-Rank Adaptation (LoRA), for clinical note classification under real-world, resource-constrained conditions. All experiments were conducted on a single NVIDIA Quadro P620 GPU (2 GB VRAM, 512 CUDA cores, 1.386 TFLOPS FP32), limiting batch sizes to <8 sequences and maximum sequence length to 256 tokens. Our clinical corpus comprises only 580 000 tokens, several orders of magnitude smaller than standard LLM pre-training datasets. We fine-tuned three biomedical pre-trained LLMs (CamemBERT-bio, AliBERT, DrBERT) and two lightweight Transformer models trained from scratch. Results show that 1) adapter structures provide no consistent gains when fine-tuning biomedical LLMs under these constraints, and 2) simpler Transformers, with minimal parameter counts and training times under six hours, outperform adapter-augmented LLMs, which required over 1000 GPU-hours. Among adapters, GRN achieved the best metrics (accuracy, precision, recall, F1 = 0.88). These findings demonstrate that, in low-resource clinical settings with limited data and compute, lightweight Transformers trained from scratch offer a more practical and efficient solution than large LLMs, while GRN remains a viable adapter choice when minimal adaptation is needed.
Accepted for publication in the IEEE Access
References in corpus (10)
- Scaling Laws for Neural Language Models
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Exploring the Limits of Language Modeling
- Training Tips for the Transformer Model
- LoRA Learns Less and Forgets Less
- Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation
- Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset
- Label Propagation Techniques for Artifact Detection in Imbalanced Classes using Photoplethysmogram Signals
- GRN-Transformer: Enhancing Motion Artifact Detection in PICU Photoplethysmogram Signals
- Transformer Meets Gated Residual Networks To Enhance Photoplethysmogram Artifact Detection Informed by Mutual Information Neural Estimation