22 citations · 33 across the 23 of their papers we have counts for
23 papers
Hunyuan-A13B Technical Report
Tencent Hunyuan Team, Ao Liu, Botong Zhou +72
We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion durin…
Dynamic Important Example Mining for Reinforcement Finetuning
Haoru Tan, Sitong Wu, Yanfeng Chen +9
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and use…
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
Songhao Wu, Ang Lv, Ruobing Xie +1
Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determi…
Hybrid Policy Distillation for LLMs
Wenhong Zhu, Ruobing Xie, Rui Wang +1
Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimiz…
OMNIFLOW: A Physics-Grounded Multimodal Agent for Generalized Scientific Reasoning
Hao Wu, Yongheng Zhang, Yuan Gao +7
Large Language Models (LLMs) have demonstrated exceptional logical reasoning capabilities but frequently struggle with the continuous spatiotemporal dynamics governed by Partial Di…
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie +3
On-policy distillation (OPD), which aligns the student with the teacher's logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving…
DQuant: Accurate Low-bit Post-Training Weight Quantization for LLMs
Xianglong Yan, ChengZhu Bao, Zhiteng Li +5
Large language models (LLMs) deliver strong performance, but their high compute and memory costs make deployment difficult in resource-constrained scenarios. Weight-only post-train…
Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models
Hengyuan Zhang, Zhihao Zhang, Mingyang Wang +26
Mechanistic Interpretability (MI) has emerged as a vital approach to demystify the opaque decision-making of Large Language Models (LLMs). However, existing reviews primarily treat…
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Hengyuan Zhang, Shiping Yang, Xiao Liang +8
Training student models on synthetic data generated by strong teacher models is a promising way to distilling the capabilities of teachers. However, recent studies show that strong…
Advanced Long-term Earth System Forecasting
Hao Wu, Yuan Gao, Ruijian Gou +30
Reliable long-term forecasting of Earth system dynamics is fundamentally limited by instabilities in current artificial intelligence (AI) models during extended autoregressive simu…
Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference
Jianxing Yu, Shiqi Wang, Han Yin +4
This paper focuses on detecting clickbait posts on the Web. These posts often use eye-catching disinformation in mixed modalities to mislead users to click for profit. That affects…
Exploring the Benefit of Activation Sparsity in Pre-training
Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin +7
Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activatio…
Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence
Weize Chen, Ziming You, Ran Li +7
The rapid advancement of large language models (LLMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often stru…
Advancing LLM Reasoning Generalists with Preference Trees
Lifan Yuan, Ganqu Cui, Hanbin Wang +12
We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve state-of-the-art results amon…
Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
Ning Ding, Yulin Chen, Ganqu Cui +6
Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that stri…
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
Yiju Guo, Ganqu Cui, Lifan Yuan +9
Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferen…
Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication
Weize Chen, Chenfei Yuan, Jiarui Yuan +6
Natural language (NL) has long been the predominant format for human cognition and communication, and by extension, has been similarly pivotal in the development and application of…
Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language
Zekai Qu, Ruobing Xie, Chaojun Xiao +5
With the thriving of pre-trained language model (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual in…
Universal Multi-modal Multi-domain Pre-trained Recommendation
Wenqi Sun, Ruobing Xie, Shuqing Bian +2
There is a rapidly-growing research interest in modeling user preferences via pre-training multi-domain interactions for recommender systems. However, Existing pre-trained multi-do…
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Xiaozhi Wang, Hao Peng, Yong Guan +9
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-eve…
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Weize Chen, Xiaoyue Xu, Xu Han +5
Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and…
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Chaojun Xiao, Yuqi Luo, Wenbin Zhang +8
Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs. In this paper, w…
UltraFeedback: Boosting Language Models with Scaled AI Feedback
Ganqu Cui, Lifan Yuan, Ning Ding +9
Learning from human feedback has become a pivot technique in aligning large language models (LLMs) with human preferences. However, acquiring vast and premium human feedback is bot…