2 citations · 13 across the 26 of their papers we have counts for
8 papers · 1 filter
On Flow Matching KL Divergence
Maojiang Su, Jerry Yao-Chieh Hu, Sophia Pi +1
We derive a deterministic, non-asymptotic upper bound on the Kullback-Leibler (KL) divergence of the flow-matching distribution approximation. In particular, if the flow-matc…
On Structured State-Space Duality
Jerry Yao-Chieh Hu, Xiwen Zhang, Ali ElSheikh +2
Structured State-Space Duality (SSD) [Dao & Gu, ICML 2024] is an equivalence between a simple Structured State-Space Model (SSM) and a masked attention mechanism. In particular, a…
A Theoretical Analysis of Discrete Flow Matching Generative Models
Maojiang Su, Mingcheng Lu, Jerry Yao-Chieh Hu +4
We provide a theoretical analysis for end-to-end training Discrete Flow Matching (DFM) generative models. DFM is a promising discrete generative modeling framework that learns the…
In-Context Algorithm Emulation in Fixed-Weight Transformers
Jerry Yao-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang +1
We prove that a minimal Transformer with frozen weights emulates a broad class of algorithms by in-context prompting. We formalize two modes of in-context algorithm emulation. In t…
Are Hallucinations Bad Estimations?
Hude Liu, Jerry Yao-Chieh Hu, Jennifer Yuntong Zhang +2
We formalize hallucinations in generative models as failures to link an estimate to any plausible cause. Under this interpretation, we show that even loss-minimizing optimal estima…
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
Jerry Yao-Chieh Hu, Xiwen Zhang, Maojiang Su +2
We study the computational limits of learning -bit Boolean functions (specifically, , , and their noisy variants), using a minimalist single-head soft…
Attention Mechanism, Max-Affine Partition, and Universal Approximation
Hude Liu, Jerry Yao-Chieh Hu, Zhao Song +1
We establish the universal approximation capability of single-layer, single-head self- and cross-attention mechanisms with minimal attached structures. Our key insight is to interp…
Universal Approximation with Softmax Attention
Jerry Yao-Chieh Hu, Hude Liu, Hong-Yu Chen +2
We prove that with linear transformations, both (i) two-layer self-attention and (ii) one-layer self-attention followed by a softmax function are universal approximators for contin…