6 papers · 1 filter
Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees
Yuchen Liang, Yingbin Liang, Lifeng Lai +1
Discrete diffusion models have recently gained significant prominence in applications involving natural language and graph data. A key factor influencing their effectiveness is the…
Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
Hongbo Li, Qinhang Wu, Sen Lin +2
Mixture-of-Experts (MoE) models improve transformer efficiency but lack a unified theoretical explanation, especially when both feed-forward and attention layers are allowed to spe…
Monitoring State Transitions in Markovian Systems with Sampling Cost
Kumar Saurav, Ness B. Shroff, Yingbin Liang
We consider a node-monitor pair, where the node's state varies with time. The monitor needs to track the node's state at all times; however, there is a fixed cost for each state qu…
Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
Lucas Carrit Delgado Pinheiro, Ziru Chen, Bruno Caixeta Piazza +4
While task-specific demonstrations show early success in applying large language models (LLMs) to automate some astronomical research tasks, they only provide incomplete views of a…
Provable In-Context Learning of Nonlinear Regression with Transformers
Hongbo Li, Lingjie Duan, Yingbin Liang
The transformer architecture, which processes sequences of input tokens to produce outputs for query tokens, has revolutionized numerous areas of machine learning. A defining featu…
Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
Junze Deng, Qinhang Wu, Peizhong Ju +3
Rehearsal-based methods have shown superior performance in addressing catastrophic forgetting in continual learning (CL) by storing and training on a subset of past data alongside…