32 papers
HP-JEPA: Hierarchical Partitioning for Multi-Resolution Graph Joint-Embedding Predictive Learning
Ruichen Xu, Jingxiang Qu, Wenhan Gao +5
Graph self-supervised learning aims to learn transferable representations from large-scale unlabeled graph data. Joint-embedding predictive architectures (JEPAs) avoid explicit neg…
Separating Representation from Reconstruction Enables Scalable Text Encoders
Megi Dervishi, Mathurin Videau, Yann LeCun
While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the r…
Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement
Nian Liu, Nikita Kazeev, Stephen Gregory Dale +8
De novo crystal generation seeks to discover materials that are not merely realistic, but also stable and novel. However, most existing generative models are trained to maximize th…
SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation
Mahi Luthra, Jiayi Shen, Maxime Poli +14
Human infants, with only a few hundred hours of speech exposure, acquire basic units of new languages, highlighting a striking efficiency gap compared to the data-hungry self-super…
The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
Shangwen Sun, Alfredo Canziani, Yann LeCun +1
We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention si…
AI+HW 2035: Shaping the Next Decade
Deming Chen, Jason Cong, Azalia Mirhoseini +27
Artificial intelligence (AI) and hardware (HW) are advancing at unprecedented rates, yet their trajectories have become inseparably intertwined. The global research community lacks…