10 papers
Esoteric Language Models: A Family of Any-Order Diffusion LLMs
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri +7
Diffusion-based language models offer a compelling alternative to autoregressive (AR) models by enabling parallel and controllable generation. Within this family, Masked Diffusion…
GQA-μP: The maximal parameterization update for grouped query attention
Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7
Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization…
EMO: Frustratingly Easy Progressive Training of Extendable MoE
Linghao Jin, Chufan Shi, Huijuan Wang +4
Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total…
LAPS: A Length-Aware-Prefill LLM Serving System
Jianshu She, Zonghang Li, Hongchao Du +7
LAPS identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to…
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
Wenhao Zheng, Yixiao Chen, Weitong Zhang +6
Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained…
Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation
Yuheng Zha, Kun Zhou, Yujia Wu +7
Despite their success, current training pipelines for reasoning VLMs focus on a limited range of tasks, such as mathematical and logical reasoning. As a result, these models face d…