31 citations · 55 across the 7 of their papers we have counts for
8 papers
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4
Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by…
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo +4
Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior gl…
ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
Zhiwei Hao, Jianyuan Guo, Li Shen +4
Recent advancements in vision transformers (ViTs) have demonstrated that larger models often achieve superior performance. However, training these models remains computationally in…
Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
Zhiwei Hao, Jianyuan Guo, Li Shen +6
Large language models (LLMs) have achieved impressive performance across various domains. However, the substantial hardware resources required for their training present a signific…
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning
Zhiwei Hao, Jianyuan Guo, Li Shen +3
Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioni…
Data-efficient Large Vision Models through Sequential Autoregression
Jianyuan Guo, Zhiwei Hao, Chengcheng Wang +5
Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended…
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
Zhiwei Hao, Jianyuan Guo, Kai Han +4
Knowledge distillation~(KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillat…
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Zhiwei Hao, Jianyuan Guo, Ding Jia +5
In the past few years, transformers have achieved promising performances on various computer vision tasks. Unfortunately, the immense inference overhead of most existing vision tra…