248 citations · 755 across the 30 of their papers we have counts for
20 papers · 1 filter
Generalized Neural Collapse for a Large Number of Classes
Jiachen Jiang, Jinxin Zhou, Peng Wang +4
Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such…
Are All Losses Created Equal: A Neural Collapse Perspective
Jinxin Zhou, Chong You, Xiao Li +4
While cross entropy (CE) is the most commonly used loss to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empiric…
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Zonglin Li, Chong You, Srinadh Bhojanapalli +8
This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the interm…
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…
On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
Jinxin Zhou, Xiao Li, Tianyu Ding +3
When training deep neural networks for classification tasks, an intriguing empirical phenomenon has been widely observed in the last-layer classifiers and features, where (i) the c…
Robust Training under Label Noise by Over-parameterization
Sheng Liu, Zhihui Zhu, Qing Qu +1
Recently, over-parameterized deep networks, with increasingly more network parameters than training samples, have dominated the performances of modern machine learning. However, wh…
A Geometric Analysis of Neural Collapse with Unconstrained Features
Zhihui Zhu, Tianyu Ding, Jinxin Zhou +4
We provide the first global optimization landscape analysis of -- an intriguing empirical phenomenon that arises in the last-layer classifiers and features of ne…
ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
Kwan Ho Ryan Chan, Yaodong Yu, Chong You +3
This work attempts to provide a plausible theoretical framework that aims to interpret modern deep (convolutional) networks from the principles of data compression and discriminati…
Incremental Learning via Rate Reduction
Ziyang Wu, Christina Baek, Chong You +1
Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes. The…
Deep Networks from the Principle of Rate Reduction
Kwan Ho Ryan Chan, Yaodong Yu, Chong You +3
This work attempts to interpret modern deep (convolutional) networks from the principles of rate reduction and (shift) invariant classification. We show that the basic iterative gr…
A Critique of Self-Expressive Deep Subspace Clustering
Benjamin D. Haeffele, Chong You, René Vidal
Subspace clustering is an unsupervised clustering technique designed to cluster data that is supported on a union of linear subspaces, with each subspace defining a cluster with di…
Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization
Chong You, Zhihui Zhu, Qing Qu +1
Recent advances have shown that implicit bias of gradient descent on over-parameterized models enables the recovery of low-rank matrices from linear measurements, even with no prio…
Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
Yaodong Yu, Kwan Ho Ryan Chan, Chong You +2
To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction ($\text{M…
Self-Representation Based Unsupervised Exemplar Selection in a Union of Subspaces
Chong You, Chi Li, Daniel P. Robinson +1
Finding a small set of representatives from an unlabeled dataset is a core problem in a broad range of applications such as dataset summarization and information extraction. Classi…
Recovery and Generalization in Over-Realized Dictionary Learning
Jeremias Sulam, Chong You, Zhihui Zhu
In over two decades of research, the field of dictionary learning has gathered a large collection of successful applications, and theoretical guarantees for model recovery are know…
Is an Affine Constraint Needed for Affine Subspace Clustering?
Chong You, Chun-Guang Li, Daniel P. Robinson +1
Subspace clustering methods based on expressing each data point as a linear combination of other data points have achieved great success in computer vision applications such as mot…
Stochastic Sparse Subspace Clustering
Ying Chen, Chun-Guang Li, Chong You
State-of-the-art subspace clustering methods are based on self-expressive model, which represents each data point as a linear combination of other data points. By enforcing such re…
Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
Zitong Yang, Yaodong Yu, Chong You +2
The classical bias-variance trade-off predicts that bias decreases and variance increase with model complexity, leading to a U-shaped risk curve. Recent work calls this into questi…
Basis Pursuit and Orthogonal Matching Pursuit for Subspace-preserving Recovery: Theoretical Analysis
Daniel P. Robinson, Rene Vidal, Chong You
Given an overcomplete dictionary and a signal for some sparse vector whose nonzero entries correspond to linearly independent columns of , classical sparse…
Oracle Based Active Set Algorithm for Scalable Elastic Net Subspace Clustering
Chong You, Chun-Guang Li, Daniel P. Robinson +1
State-of-the-art subspace clustering methods are based on expressing each data point as a linear combination of other data points while regularizing the matrix of coefficients with…