22 citations · 107 across the 37 of their papers we have counts for
16 papers · 1 filter
Learning Functional Subspaces for Neural Network Compression
Massimo Bini, Anders Christensen, Stephan Alaniz +5
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus…
SCOPE: Observation-Conditioned Full-Target Prediction for Sparse PDE Inference
Ruichen Xu, Siyao Wang, Fang Wan +8
Recovering complete physical fields from sparse observations is challenging because the measurements may not uniquely determine the underlying state. Diffusion-based PDE solvers ad…
PDE-OBS: Controlled Evaluation Across Observation Patterns
Ruichen Xu, Siyao Wang, Fang Wan +7
Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern does not characterize perform…
HP-JEPA: Hierarchical Partitioning for Multi-Resolution Graph Joint-Embedding Predictive Learning
Ruichen Xu, Jingxiang Qu, Wenhan Gao +5
Graph self-supervised learning aims to learn transferable representations from large-scale unlabeled graph data. Joint-embedding predictive architectures (JEPAs) avoid explicit neg…
Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement
Nian Liu, Nikita Kazeev, Stephen Gregory Dale +8
De novo crystal generation seeks to discover materials that are not merely realistic, but also stable and novel. However, most existing generative models are trained to maximize th…
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Hai Huang, Yann LeCun, Randall Balestriero
Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these la…
Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
Enrique Queipo-de-Llano, Álvaro Arroyo, Federico Barbero +4
Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, w…
Transformers without Normalization
Jiachen Zhu, Xinlei Chen, Kaiming He +2
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve th…
Layer by Layer: Uncovering Hidden Representations in Language Models
Oscar Skean, Md Rifat Arefin, Dan Zhao +4
From extracting features to generating text, the outputs of large language models (LLMs) typically rely on the final layers, following the conventional wisdom that earlier layers c…
Does Representation Matter? Exploring Intermediate Layers in Large Language Models
Oscar Skean, Md Rifat Arefin, Yann LeCun +1
Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we inve…
Rate-In: Information-Driven Adaptive Dropout Rates for Improved Inference-Time Uncertainty Estimation
Tal Zeevi, Ravid Shwartz-Ziv, Yann LeCun +2
Accurate uncertainty estimation is crucial for deploying neural networks in risk-sensitive applications such as medical diagnosis. Monte Carlo Dropout is a widely used technique fo…
Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization
Deep Chakraborty, Yann LeCun, Tim G. J. Rudner +1
A number of different architectures and loss functions have been applied to the problem of self-supervised learning (SSL), with the goal of developing embeddings that provide the b…
Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier +4
Decoder-only Transformers often struggle with complex reasoning tasks, particularly arithmetic reasoning requiring multiple sequential operations. In this work, we identify represe…
Just How Flexible are Neural Networks in Practice?
Ravid Shwartz-Ziv, Micah Goldblum, Arpit Bansal +3
It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters, underpinning notions of overparameterized and underpara…
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai +10
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is intri…
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
Xiaoxin He, Yijun Tian, Yifei Sun +5
Given a graph with textual attributes, we enable users to `chat with their graph': that is, to ask questions about the graph using a conversational interface. In response to a user…