2 citations · 2 across the 5 of their papers we have counts for
6 papers
Contrastive On-Policy Distillation
Jiacheng Ruan, Jun Tang, Wenzhen Yuan +5
On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and…
MMGist: A Comprehensive Multimodal Benchmark for 2027
Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3
We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effective…
ExFusion: Efficient Transformer Training via Multi-Experts Fusion
Jiacheng Ruan, Daize Dong, Xiaoye Qu +5
Mixture-of-Experts (MoE) models substantially improve performance by increasing the capacity of dense architectures. However, directly training MoE models requires considerable com…
FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion
Jiacheng Ruan, Yebin Yang, Zehao Lin +4
Benefiting from the revolutionary advances in large language models (LLMs) and foundational vision models, large vision-language models (LVLMs) have also made significant progress.…
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Tong Zhu, Xiaoye Qu, Daize Dong +4
Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale…
Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts
Tong Zhu, Daize Dong, Xiaoye Qu +3
Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all train…