3 papers
cs.CL2026
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Ziyang Cai, Xingyu Zhu, Yihe Dong +2
The paper proposes T²MLR, a transformer variant that injects a cached middle‑layer representation from the previous token into an earlier layer of the current token, allowing inter…
cs.LG2026
Muon: Muon with Fractional Spectral Powers
Yihe Dong, Will Sawin
Muon is an increasingly widely used optimizer that replaces a gradient with its polar factor , thereby flattening the singular spectrum. However, full flatten…
cs.LG2025
Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
Yihe Dong, Lorenzo Noci, Mikhail Khodak +1
The transformer architecture is central to the success of modern Large Language Models (LLMs), in part due to its surprising ability to perform a wide range of tasks - including ma…