5 papers
Scaling Native Multimodal Pre-Training From Scratch
Haoyuan Wu, Aoqi Wu, Hai Wang +3
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world.…
Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
Jinrui Zhang, Chaodong Xiao, Aoqi Wu +2
Pretraining large language models (LLMs) typically requires centralized clusters with thousands of high-memory GPUs (e.g., H100/A100). Recent decentralized training methods reduce…
Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
Junming Liu, Yanting Gao, Yuqi Li +6
Federated Learning (FL) is a decentralized machine learning paradigm that enables clients to collaboratively train models while preserving data privacy. However, the coexistence of…
LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
Weiquan Huang, Aoqi Wu, Yifan Yang +10
CLIP is a seminal multimodal model that maps images and text into a shared representation space through contrastive learning on billions of image-caption pairs. Inspired by the rap…
D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
Evelyn Zhang, Fufu Yu, Aoqi Wu +5
Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we f…