Showing 2024Show all
2 papers · 1 filter
cs.CL2024
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Xiaolin Hu, Xiang Cheng, Peiyu Liu +4
Low-rank adaptation (LoRA) reduces the computational and memory demands of fine-tuning large language models (LLMs) by approximating updates with low-rank matrices. However, low-ra…
cs.LG2024
Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Xinhao Yao, Xiaolin Hu, Shenzhi Yang +1
Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can p…