22 papers
Spokes: Optimizing for Diverse Pretraining Data Selection
Clarence Lee, Yejin Choi, Luke Zettlemoyer +2
Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inhere…
Scaling Participation in Modular AI Systems
Shangbin Feng, Yike Wang, Weijia Shi +3
Humanity is a mosaic of multifaceted talents and needs, and any truly intelligent AI must reflect that richness. Yet the LLMs used by all are built by the few -- a centralized mark…
JobBench: Aligning Agent Work With Human Will
Yuetai Li, Yichen Feng, Zhangchen Xu +21
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflow…
Slicing and Dicing: Configuring Optimal Mixtures of Experts
Margaret Li, Sneha Kudugunta, Danielle Rothermel +1
Mixture-of-Experts (MoE) architectures have become standard in large language models, yet many of their core design choices - expert count, granularity, shared experts, load balanc…
Micro Language Models Enable Instant Responses
Wen Cheng, Tuochao Chen, Karim Helwani +3
Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter language models due to power and compute constraints, yet cloud infe…
MoCo: A One-Stop Shop for Model Collaboration Research
Shangbin Feng, Yuyang Bai, Ziyuan Yang +17
Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, an…