2 citations · 7 across the 33 of their papers we have counts for
4 papers · 1 filter
Multimodal Latent Language Modeling with Next-Token Diffusion
Yutao Sun, Hangbo Bao, Wenhui Wang +5
Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video). In this work, we propose…
RedStone: Curating General, Code, Math, and QA Data for Large Language Models
Yaoyao Chang, Lei Cui, Li Dong +13
Pre-training Large Language Models (LLMs) on high-quality, meticulously curated datasets is widely recognized as critical for enhancing their performance and generalization capabil…
Data Selection via Optimal Control for Language Models
Yuxian Gu, Li Dong, Hongning Wang +4
This work investigates the selection of high-quality pre-training data from massive corpora to enhance LMs' capabilities for downstream usage. We formulate data selection as a gene…
Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
Jiawei Zhou, Li Dong, Furu Wei +1
Information retrieval has transitioned from standalone systems into essential components across broader applications, with indexing efficiency, cost-effectiveness, and freshness be…