5 papers
GQA-μP: The maximal parameterization update for grouped query attention
Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7
Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization…
EMO: Frustratingly Easy Progressive Training of Extendable MoE
Linghao Jin, Chufan Shi, Huijuan Wang +4
Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total…
From Reasoning to Pixels: Benchmarking the Alignment Gap in Unified Multimodal Models
Cheng Yang, Chufan Shi, Bo Shui +7
Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent their representations…
Asymmetric Idiosyncrasies in Multimodal Models
Muzi Tao, Chufan Shi, Huijuan Wang +2
In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption o…
K2-V2: A 360-Open, Reasoning-Enhanced LLM
K2 Team, Zhengzhong Liu, Liping Tang +36
We introduce K2-V2, a 360-open LLM built from scratch as a superior base for reasoning adaptation, in addition to functions such as conversation and knowledge retrieval from genera…