3 citations · 7 across the 15 of their papers we have counts for
5 papers · 1 filter
LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
Runming Yang, Taiqiang Wu, Jiahao Wang +4
Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstr…
Autoregressive Models in Vision: A Survey
Jing Xiong, Gongye Liu, Lun Huang +17
Autoregressive modeling has been a huge success in the field of natural language processing (NLP). Recently, autoregressive models have emerged as a significant area of focus in co…
A Survey on the Honesty of Large Language Models
Siheng Li, Cheng Yang, Taiqiang Wu +12
Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to fai…
Mixture-of-Subspaces in Low-Rank Adaptation
Taiqiang Wu, Jiahao Wang, Zhe Zhao +1
In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language,…
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
Taiqiang Wu, Chaofan Tao, Jiahao Wang +3
Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler…