HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation

Anchun Gui, Han Xiao

To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire param…