164 citations · 356 across the 41 of their papers we have counts for
28 papers · 1 filter
Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules
Florian Rottach, Sebastian Schieferdecker, William Rudman +2
Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of bi…
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Hai Huang, Yann LeCun, Randall Balestriero
Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these la…
Radial-VCReg: More Informative Representation Learning Through Radial Gaussianization
Yilun Kuang, Yash Dagade, Deep Chakraborty +4
Self-supervised learning aims to learn maximally informative representations, but explicit information maximization is hindered by the curse of dimensionality. Existing methods lik…
Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations
Yilun Kuang, Yash Dagade, Tim G. J. Rudner +2
Joint-Embedding Predictive Architectures (JEPA) learn view-invariant representations and admit projection-based distribution matching for collapse prevention. Existing approaches r…
LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
Randall Balestriero, Yann LeCun
Learning manipulable representations of the world and its dynamics is central to AI. Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of pract…
Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density
Randall Balestriero, Nicolas Ballas, Mike Rabbat +1
Joint Embedding Predictive Architectures (JEPAs) learn representations able to solve numerous downstream tasks out-of-the-box. JEPAs combine two objectives: (i) a latent-space pred…
Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
Hugues Van Assel, Mark Ibrahim, Tommaso Biancalani +2
Reconstruction and joint embedding have emerged as two leading paradigms in Self Supervised Learning (SSL). Reconstruction methods focus on recovering the original sample from a di…
Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Vlad Sobal, Wancong Zhang, Kyunghyun Cho +3
A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms…
Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
Mark Ibrahim, David Klindt, Randall Balestriero
Deep Learning is often depicted as a trio of data-architecture-loss. Yet, recent Self Supervised Learning (SSL) solutions have introduced numerous additional design choices, e.g.,…
Fast and Exact Enumeration of Deep Networks Partitions Regions
Randall Balestriero, Yann LeCun
One fruitful formulation of Deep Networks (DNs) enabling their theoretical study and providing practical guidelines to practitioners relies on Piecewise Affine Splines. In that rea…
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Florian Bordes, Samuel Lavoie, Randall Balestriero +2
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations. Because this pretext task differs from the downstream tasks used to evaluate the performance…
A Cookbook of Self-Supervised Learning
Randall Balestriero, Mark Ibrahim, Vlad Sobal +16
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art wi…
Towards Democratizing Joint-Embedding Self-Supervised Learning
Florian Bordes, Randall Balestriero, Pascal Vincent
Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data. The development of J…
Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need
Vivien Cabannes, Leon Bottou, Yann Lecun +1
Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. However, SSL requires to build samples that are know…
On minimal variations for unsupervised representation learning
Vivien Cabannes, Alberto Bietti, Randall Balestriero
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learni…
POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks
Randall Balestriero, Yann LeCun
Deep Neural Networks (DNNs) outshine alternative function approximators in many settings thanks to their modularity in composing any desired differentiable operator. The formed par…
The Hidden Uniform Cluster Prior in Self-Supervised Learning
Mahmoud Assran, Randall Balestriero, Quentin Duval +6
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show tha…
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank
Quentin Garrido, Randall Balestriero, Laurent Najman +1
Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations but only few principled guidelines that would help prac…
Joint Embedding Self-Supervised Learning in the Kernel Regime
Bobak T. Kiani, Randall Balestriero, Yubei Chen +2
The fundamental goal of self-supervised learning (SSL) is to produce useful representations of data without access to any labels for classifying the data. Modern methods in SSL, wh…
Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
Grégoire Mialon, Randall Balestriero, Yann LeCun
Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance ma…
What Do We Maximize in Self-Supervised Learning?
Ravid Shwartz-Ziv, Randall Balestriero, Yann LeCun
In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction. As a first step, we demo…
Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
Randall Balestriero, Yann LeCun
Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a mileston…
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
Florian Bordes, Randall Balestriero, Quentin Garrido +2
One unexpected technique that emerged in recent years consists in training a Deep Network (DN) with a Self-Supervised Learning (SSL) method, and using this network on downstream ta…
The Effects of Regularization and Data Augmentation are Class Dependent
Randall Balestriero, Leon Bottou, Yann LeCun
Regularization is a fundamental technique to prevent over-fitting and to improve generalization performances by constraining a model's complexity. Current Deep Networks heavily rel…