29 citations · 36 across the 16 of their papers we have counts for
Showing cs.LGShow all
2 papers · 1 filter
cs.LG2024
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Chuanyu Tang, Yilong Chen, Zhenyu Zhang +4
Low-Rank Adaptation (LoRA) drives research to align its performance with full fine-tuning. However, significant challenges remain: (1) Simply increasing the rank size of LoRA does…
cs.LG2024
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Yilong Chen, Linhao Zhang, Junyuan Shang +4
Large language models (LLMs) with billions of parameters demonstrate impressive performance. However, the widely used Multi-Head Attention (MHA) in LLMs incurs substantial computat…