6 papers
A Generalization Theory for JEPA-Based World Models
Jingyi Cui, Qi Zhang, Hongwei Wen +1
Joint Embedding Predictive Architectures (JEPAs) have recently emerged as a promising paradigm for world modeling by learning predictive dynamics in a latent space rather than gene…
A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima
Yiming Tang, Harshvardhan Saini, Zhaoqian Yao +6
As AI models achieve remarkable capabilities across diverse domains, understanding what representations they learn and how they encode concepts has become increasingly important fo…
On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy
Jingyi Cui, Qi Zhang, Yifei Wang +1
Sparse autoencoders (SAEs) have recently emerged as a powerful tool for interpreting the features learned by large language models (LLMs). By reconstructing features with sparsely…
Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective
Yi-Ge Zhang, Jingyi Cui, Qiran Li +1
Unsupervised contrastive learning has shown significant performance improvements in recent years, often approaching or even rivaling supervised learning in various tasks. However,…
An Augmentation-Aware Theory for Self-Supervised Contrastive Learning
Jingyi Cui, Hongwei Wen, Yisen Wang
Self-supervised contrastive learning has emerged as a powerful tool in machine learning and computer vision to learn meaningful representations from unlabeled data. Meanwhile, its…
An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise
Jingyi Cui, Yi-Ge Zhang, Hengyu Liu +1
Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently…