5 papers · 1 filter
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Mind Lab, :, Vin Bo +64
Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state…
AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning
Changhai Zhou, Shiyang Zhang, Yuhua Zhou +5
Quantization followed by parameter-efficient fine-tuning has emerged as a promising paradigm for downstream adaptation under tight GPU memory constraints. However, this sequential…
Shared Nature, Unique Nurture: PRISM for Pluralistic Reasoning via In-context Structure Modeling
Guancheng Tu, Shiyang Zhang, Tianyu Zhang +2
Large Language Models (LLMs) are converging towards a singular Artificial Hivemind, where shared Nature (pre-training priors) result in a profound collapse of distributional divers…
Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies
Changhai Zhou, Shiyang Zhang, Yuhua Zhou +5
Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. Wh…
AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning
Changhai Zhou, Shiyang Zhang, Yuhua Zhou +2
Fine-tuning large language models (LLMs) under resource constraints is a significant challenge in deep learning. Low-Rank Adaptation (LoRA), pruning, and quantization are all effec…