23 citations · 72 across the 17 of their papers we have counts for
17 papers
Attention-Only Transformers via Unrolled Subspace Denoising
Peng Wang, Yifu Lu, Yaodong Yu +3
Despite the popularity of transformers in practice, their architectures are empirically designed and neither mathematically justified nor interpretable. Moreover, as indicated by m…
Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction
Ziyang Wu, Tianjiao Ding, Yifu Lu +6
The attention operator is arguably the key distinguishing factor of transformer architectures, which have demonstrated state-of-the-art performance on a variety of tasks. However,…
A Global Geometric Analysis of Maximal Coding Rate Reduction
Peng Wang, Huikang Liu, Druv Pai +4
The maximal coding rate reduction (MCR) objective for learning structured and compact deep representations is drawing increasing attention, especially after its recent usage in…
Scaling White-Box Transformers for Vision
Jinrui Yang, Xianhang Li, Druv Pai +4
CRATE, a white-box transformer architecture designed to learn compressed and sparse representations, offers an intriguing alternative to standard vision transformers (ViTs) due to…
Masked Completion via Structured Diffusion with White-Box Transformers
Druv Pai, Ziyang Wu, Sam Buchanan +2
Modern learning frameworks often train deep neural networks with massive amounts of unlabeled data to learn representations by solving simple pretext tasks, then use the representa…
Differentially Private Representation Learning via Image Captioning
Tom Sander, Yaodong Yu, Maziar Sanjabi +4
Differentially private (DP) machine learning is considered the gold-standard solution for training a model from sensitive data while still preserving privacy. However, a major barr…
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
Yaodong Yu, Sam Buchanan, Druv Pai +7
In this paper, we contend that a natural objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a low-dimension…
Emergence of Segmentation with Minimalistic White-Box Transformers
Yaodong Yu, Tianzhe Chu, Shengbang Tong +4
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection. Previous works have shown th…
Scaff-PD: Communication Efficient Fair and Robust Federated Learning
Yaodong Yu, Sai Praneeth Karimireddy, Yi Ma +1
We present Scaff-PD, a fast and communication-efficient algorithm for distributionally robust federated learning. Our approach improves fairness by optimizing a family of distribut…
ViP: A Differentially Private Foundation Model for Computer Vision
Yaodong Yu, Maziar Sanjabi, Yi Ma +2
Artificial intelligence (AI) has seen a tremendous surge in capabilities thanks to the use of foundation models trained on internet-scale data. On the flip side, the uncurated natu…
White-Box Transformers via Sparse Rate Reduction
Yaodong Yu, Sam Buchanan, Druv Pai +5
In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a mixture of low-dime…
Robust Calibration with Multi-domain Temperature Scaling
Yaodong Yu, Stephen Bates, Yi Ma +1
Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains. Uncertainty quantification is all the more challe…
Conditional Supervised Contrastive Learning for Fair Text Classification
Jianfeng Chi, William Shand, Yaodong Yu +3
Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data. However, the learned…
Predicting Out-of-Distribution Error with the Projection Norm
Yaodong Yu, Zitong Yang, Alexander Wei +2
We propose a metric -- Projection Norm -- to predict a model's performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model…
The Effect of Model Size on Worst-Group Generalization
Alan Pham, Eunice Chan, Vikranth Srivatsa +6
Overparameterization is shown to result in poor test accuracy on rare subgroups under a variety of settings where subgroup information is known. To gain a more complete picture, we…
Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
Xili Dai, Shengbang Tong, Mingyang Li +8
This work proposes a new computational framework for learning a structured generative model for real-world datasets. In particular, we propose to learn a closed-loop transcription…
On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averaging
Chris Junchi Li, Yaodong Yu, Nicolas Loizou +4
We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) method with constant step size, and presenti…