12 citations · 15 across the 12 of their papers we have counts for
14 papers
ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
Zichun Yu, Jiarui Yan, Shlok Sanghvi +2
LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then c…
Spectral Guidance for Flexible and Efficient Control of Diffusion Models
Gabriel Moreira, Manuel Marques, João Paulo Costeira +1
We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by n…
Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
Zichun Yu, Chenyan Xiong
LLM pretraining is shifting from a compute-bound to a data-bound regime, where available human (organic) text falls far short of scaling demands. However, reaching the data-bound r…
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Shanshan Zhong, Yi Lu, Jingjie Ning +7
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically…
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
Yiyang Du, Zhanqiu Guo, Haochen Zhang +5
Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are n…
Midtraining Bridges Pretraining and Posttraining Distributions
Emmy Liu, Graham Neubig, Chenyan Xiong
Midtraining, the practice of mixing specialized data with more general pretraining data in an intermediate training phase, has become widespread in language model development, yet…
RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
Zichun Yu, Chenyan Xiong
High-quality data is a cornerstone of large language model (LLM) pretraining, yet its growth has not kept pace with the needs of frontier models. In this paper, we introduce RePro,…
FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
Hao Kang, Zichun Yu, Chenyan Xiong
Recent large language models such as Gemini-1.5, DeepSeek-V3, and Llama-4 increasingly adopt Mixture-of-Experts (MoE) architectures, which offer strong efficiency-performance trade…
Group-Level Data Selection for Efficient Pretraining
Zichun Yu, Fei Peng, Jie Lei +3
In this paper, we introduce Group-MATES, an efficient group-level data selection approach to optimize the speed-quality frontier of language model pretraining. Specifically, Group-…
Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search
Wentao Shi, Zichun Yu, Fuli Feng +2
Monte Carlo Tree Search (MCTS) based methods provide promising approaches for generating synthetic data to enhance the self-training of Large Language Model (LLM) based multi-agent…
Harnessing Webpage UIs for Text-Rich Visual Understanding
Junpeng Liu, Tianyue Ou, Yifan Song +6
Text-rich visual understanding-the ability to process environments where dense textual content is integrated with visuals-is crucial for multimodal large language models (MLLMs) to…
Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning
Xiaochuan Li, Zichun Yu, Chenyan Xiong
Synthetic data has been widely used to train large language models, but their generative nature inevitably introduces noisy, non-informative, and misleading learning signals. In th…
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
Zichun Yu, Spandan Das, Chenyan Xiong
Pretraining data selection has the potential to improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection…
An In-depth Look at Gemini's Language Abilities
Syeda Nahida Akter, Zichun Yu, Aashiq Muhamed +6
The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper,…