5 papers
Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent
Bo Chen, Zhenmei Shi, Zhao Song +1
Recent advancements in Transformer-based architectures have led to impressive breakthroughs in natural language processing tasks, with models such as GPT-4, Claude, and Gemini demo…
Visual Autoregressive Transformers Must Use Memory
Yang Cao, Xiaoyu Li, Yekun Ke +3
A fundamental challenge in Visual Autoregressive models is the substantial memory overhead required during inference to store previously generated representations. Despite various…
Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modeling
Yang Cao, Bo Chen, Xiaoyu Li +5
This paper introduces Force Matching (ForM), a novel framework for generative modeling that represents an initial exploration into leveraging special relativistic mechanics to enha…
High-Order Matching for One-Step Shortcut Diffusion Models
Bo Chen, Chengyue Gong, Xiaoyu Li +5
One-step shortcut diffusion models [Frans, Hafner, Levine and Abbeel, ICLR 2025] have shown potential in vision generation, but their reliance on first-order trajectory supervision…
Circuit Complexity Bounds for RoPE-based Transformer Architecture
Bo Chen, Xiaoyu Li, Yingyu Liang +3
Characterizing the express power of the Transformer architecture is critical to understanding its capacity limits and scaling law. Recent works provide the circuit complexity bound…