3 citations · 17 across the 24 of their papers we have counts for
13 papers · 1 filter
Simple-OPD: Demystifying Warm-up for On-policy Distillation
Tao Liu, Taiqiang Wu, Mao Zheng +5
On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage b…
ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
Tao Liu, Taiqiang Wu, Runming Yang +3
Supervised fine-tuning (SFT) is a fundamental post-training strategy to align Large Language Models (LLMs) with human intent. However, traditional SFT often ignores the one-to-many…
LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
Runming Yang, Taiqiang Wu, Jiahao Wang +4
Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstr…
A Survey on the Honesty of Large Language Models
Siheng Li, Cheng Yang, Taiqiang Wu +12
Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to fai…
LoCa: Logit Calibration for Knowledge Distillation
Runming Yang, Taiqiang Wu, Yujiu Yang
Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the ou…
Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
Chufan Shi, Cheng Yang, Xinyu Zhu +6
Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the input sequence activat…
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
Taiqiang Wu, Chaofan Tao, Jiahao Wang +3
Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler…
Recouple Event Field via Probabilistic Bias for Event Extraction
Xingyu Bai, Taiqiang Wu, Han Guo +7
Event Extraction (EE), aiming to identify and classify event triggers and arguments from event mentions, has benefited from pre-trained language models (PLMs). However, existing PL…
Weight-Inherited Distillation for Task-Agnostic BERT Compression
Taiqiang Wu, Cheng Hou, Shanshan Lao +4
Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the b…
SynGen: A Syntactic Plug-and-play Module for Generative Aspect-based Sentiment Analysis
Chengze Yu, Taiqiang Wu, Jiayi Li +2
Aspect-based Sentiment Analysis (ABSA) is a sentiment analysis task at fine-grained level. Recently, generative frameworks have attracted increasing attention in ABSA due to their…
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities
Zhe Zhao, Yudong Li, Cheng Hou +23
Recently, the success of pre-training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-training models of different modalities a…
Modeling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrieval
Taiqiang Wu, Xingyu Bai, Weigang Guo +3
Zero-shot entity retrieval, aiming to link mentions to candidate entities under the zero-shot setting, is vital for many tasks in Natural Language Processing. Most existing methods…
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching
Kunbo Ding, Weijie Liu, Yuejian Fang +3
Previous studies have proved that cross-lingual knowledge distillation can significantly improve the performance of pre-trained models for cross-lingual similarity matching tasks.…