1 citations · 1 across the 4 of their papers we have counts for
5 papers
Learning When Not to Attend Globally
Xuan Luo, Kailai Zhang, Xifeng Yan
When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs…
Train a Unified Multimodal Data Quality Classifier with Synthetic Data
Weizhi Wang, Rongmei Lin, Shiyang Li +7
The Multimodal Large Language Models (MLLMs) are continually pre-trained on a mixture of image-text caption data and interleaved document data, while the high-quality data filterin…
Direct Multi-Token Decoding
Xuan Luo, Weizhi Wang, Xifeng Yan
Decoder-only transformers have become the standard architecture for large language models (LLMs) due to their strong performance. Recent studies suggest that, in pre-trained LLMs,…
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources
Weizhi Wang, Yu Tian, Linjie Yang +2
The reproduction of state-of-the-art multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strat…
Adaptive Layer-skipping in Pre-trained LLMs
Xuan Luo, Weizhi Wang, Xifeng Yan
Various layer-skipping methods have been proposed to accelerate token generation in large language models (LLMs). However, limited attention has been paid to a fundamental question…