1 citations · 1 across the 8 of their papers we have counts for
9 papers
Full-bandwidth transformer
Xi Wang, Ziyang Cai, Zheng Zhan +5
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Ziyang Cai, Xingyu Zhu, Yihe Dong +2
Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reason…
Goedel-Architect: Streamlining Formal Theorem Proving with Blueprint Generation and Refinement
Jui-Hui Chung, Ziyang Cai, Zihao Li +14
We introduce Goedel-Architect, an agentic framework for formal theorem proving in Lean 4 centered on blueprint generation and refinement. A blueprint is a dependency graph of defin…
ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments
Ziyang Cai, Amir Saeidi, Harkirat Behl
With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labo…
h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
Sumeet Ramesh Motwani, Alesia Ivanova, Ziyang Cai +5
Large language models excel at short-horizon reasoning tasks, but performance drops as reasoning horizon lengths increase. Existing approaches to combat this rely on inference-time…
Extrapolation by Association: Length Generalization Transfer in Transformers
Ziyang Cai, Nayoung Lee, Avi Schwarzschild +2
Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalizatio…
R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
Albert Ge, Tzu-Heng Huang, John Cooper +7
Data mixing strategies have successfully reduced the costs involved in training language models. While promising, such methods suffer from two flaws. First, they rely on predetermi…
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
Nayoung Lee, Ziyang Cai, Avi Schwarzschild +2
Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach whe…
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition
Zheyang Xiong, Ziyang Cai, John Cooper +11
Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform…