15 citations · 33 across the 33 of their papers we have counts for
21 papers · 1 filter
Transformer Approximations from ReLUs
Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1
We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yiel…
Discrete Flow Matching Policy Optimization
Maojiang Su, Po-Chung Hsieh, Weimin Wu +4
We introduce Discrete flow Matching policy Optimization (DoMinO), a unified framework for Reinforcement Learning (RL) fine-tuning Discrete Flow Matching (DFM) models under a broad…
On Structured State-Space Duality
Jerry Yao-Chieh Hu, Xiwen Zhang, Ali ElSheikh +2
Structured State-Space Duality (SSD) [Dao & Gu, ICML 2024] is an equivalence between a simple Structured State-Space Model (SSM) and a masked attention mechanism. In particular, a…
A Theoretical Analysis of Discrete Flow Matching Generative Models
Maojiang Su, Mingcheng Lu, Jerry Yao-Chieh Hu +4
We provide a theoretical analysis for end-to-end training Discrete Flow Matching (DFM) generative models. DFM is a promising discrete generative modeling framework that learns the…
In-Context Algorithm Emulation in Fixed-Weight Transformers
Jerry Yao-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang +1
We prove that a minimal Transformer with frozen weights emulates a broad class of algorithms by in-context prompting. We formalize two modes of in-context algorithm emulation. In t…
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
Jerry Yao-Chieh Hu, Xiwen Zhang, Maojiang Su +2
We study the computational limits of learning -bit Boolean functions (specifically, , , and their noisy variants), using a minimalist single-head soft…
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Haozheng Luo, Chenghao Qiu, Maojiang Su +5
To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptabil…
Attention Mechanism, Max-Affine Partition, and Universal Approximation
Hude Liu, Jerry Yao-Chieh Hu, Zhao Song +1
We establish the universal approximation capability of single-layer, single-head self- and cross-attention mechanisms with minimal attached structures. Our key insight is to interp…
Universal Approximation with Softmax Attention
Jerry Yao-Chieh Hu, Hude Liu, Hong-Yu Chen +2
We prove that with linear transformations, both (i) two-layer self-attention and (ii) one-layer self-attention followed by a softmax function are universal approximators for contin…
Pareto-Optimal Energy Alignment for Designing Nature-Like Antibodies
Yibo Wen, Chenwei Xu, Jerry Yao-Chieh Hu +2
We present a three-stage framework for training deep learning models specializing in antibody sequence-structure co-design. We first pre-train a language model using millions of an…
In-Context Deep Learning via Transformer Models
Weimin Wu, Maojiang Su, Jerry Yao-Chieh Hu +2
We investigate the transformer's capability to simulate the training process of deep models via in-context learning (ICL), i.e., in-context deep learning. Our key contribution is p…
Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
Jerry Yao-Chieh Hu, Wei-Po Wang, Ammar Gilani +3
We investigate the statistical and computational limits of prompt tuning for transformer-based foundation models. Our key contributions are prompt tuning on \emph{single-head} tran…
Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models
Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo +2
We study the computational limits of Low-Rank Adaptation (LoRA) for finetuning transformer-based models using fine-grained complexity theory. Our key observation is that the existe…
BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model
Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu +4
We introduce the \textbf{B}i-Directional \textbf{S}parse \textbf{Hop}field Network (\textbf{BiSHop}), a novel end-to-end framework for deep tabular learning. BiSHop handles the two…
Outlier-Efficient Hopfield Layers for Large Transformer-Based Models
Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo +4
We introduce an Outlier-Efficient Modern Hopfield Model (termed ) and use it to address the outlier inefficiency problem of {training} gigantic transformer-base…
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models
Dennis Wu, Jerry Yao-Chieh Hu, Teng-Yun Hsiao +1
We propose a two-stage memory retrieval dynamics for modern Hopfield models, termed , with enhanced memory capacity. Our key contribution is a learnable feat…
On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis
Jerry Yao-Chieh Hu, Thomas Lin, Zhao Song +1
We investigate the computational limits of the memory retrieval dynamics of modern Hopfield models from the fine-grained complexity analysis. Our key contribution is the characteri…
Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e
Chenwei Xu, Jerry Yao-Chieh Hu, Aakaash Narayanan +21
We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron…
STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction
Dennis Wu, Jerry Yao-Chieh Hu, Weijian Li +2
We present STanHop-Net (Sparse Tandem Hopfield Network) for multivariate time series prediction with memory-enhanced capabilities. At the heart of our approach is STanHop, a novel…
On Sparse Modern Hopfield Model
Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu +3
We introduce the sparse modern Hopfield model as a sparse extension of the modern Hopfield model. Like its dense counterpart, the sparse modern Hopfield model equips a memory-retri…
Feature Programming for Multivariate Time Series Prediction
Alex Reneau, Jerry Yao-Chieh Hu, Chenwei Xu +3
We introduce the concept of programmable feature engineering for time series modeling and propose a feature programming framework. This framework generates large amounts of predict…