From the 1 of 50 linked papers with an AI index.
11 papers · 1 filter
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
Guanghui Wang, Kaiwen Lv Kacuila, Zhiyong Yang +5
Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teac…
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
Xingyu Lyu, Qianqian Xu, Zhiyong Yang +2
Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, ye…
Localize and Neutralize: Gradient-guided Token Suppression against Visual Prompt Injection Attack
Dongpeng Zhang, Ke Ma, Yangbangyan Jiang +4
Adversarial images pose a severe security threat to multimodal large language models through prompt injection. Existing defenses largely lack a principled understanding of the unde…
DirMixE: Harnessing Test Agnostic Long-tail Recognition with Hierarchical Label Variations
Zhiyong Yang, Qianqian Xu, Sicong Li +3
This paper explores test-agnostic long-tail recognition, a challenging long-tail task where the test label distributions are unknown and arbitrarily imbalanced. We argue that the v…
TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
Qifeng Lei, Zhiyong Yang, Qianqian Xu +3
Efficiently fine-tuning pre-trained models for downstream tasks is a key challenge in the era of foundation models. Parameter-efficient fine-tuning (PEFT) presents a promising solu…
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via --Divergence
Guanghui Wang, Zhiyong Yang, Zitai Wang +3
Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student model by minimizing the divergence between their output distributions, typically usi…