Stefanie Jegelka

How Do Transformers Learn to Represent Symmetries?

Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1

Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning. Despite its empirical success, the…

Training-Free Transformer Merging via Sequential Local Operator Alignment

Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2

Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…

Efficient Learning and Symmetry Discovery under Exact Invariances

Ashkan Soleymani, Behrooz Tahmasebi, Patrick Jaillet +1

Learning with group invariances is central to many scientific and geometric learning problems, yet its computational foundations remain poorly understood. Even for classical superv…

Data Augmentation: A Fourier Analysis Perspective

Behrooz Tahmasebi, Melanie Weber, Stefanie Jegelka

Data augmentation is a simple and model-agnostic approach for exploiting known invariances in learning problems. Given a group acting on the input space, one augments the training…

Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2

Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL…

Scaling Attention via Feature Sparsity

Yan Xie, Tiansheng Wen, Tangda Huang +4

Scaling Transformers to ultra-long contexts is bottlenecked by the $O (n^{2} d)$ cost of self-attention. Existing methods reduce this cost along the sequence axis through local window…

CSRv2: Unlocking Ultra-Sparse Embeddings

Lixuan Guo, Yifei Wang, Tiansheng Wen +5

In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. Yet widely used dens…

Route Experts by Sequence, not by Token

Tiansheng Wen, Yifei Wang, Aosong Feng +7

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed num…

LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs

Ang Li, Yifei Wang, Zhihang Yuan +2

Reinforcement learning in large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts, forcing the model…

Geometric Algorithms for Neural Combinatorial Optimization with Constraints

Nikolaos Karalias, Akbar Rafiey, Yifei Xu +4

Self-Supervised Learning (SSL) for Combinatorial Optimization (CO) is an emerging paradigm for solving combinatorial problems using neural networks. In this paper, we address a cen…

LieAugmenter: Equivariant Learning by Discovering Symmetries with Learnable Augmentations

Eduardo Santos-Escriche, Ya-Wei Eileen Lin, Stefanie Jegelka

Data augmentation is a powerful mechanism in equivariant machine learning, encouraging symmetry by training networks to produce consistent outputs under transformed inputs. Yet, ef…

G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning

Xiaojun Guo, Ang Li, Yifei Wang +2

Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly ge…

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

Tiansheng Wen, Yifei Wang, Zequn Zeng +7

Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learn…

Learning with Exact Invariances in Polynomial Time

Ashkan Soleymani, Behrooz Tahmasebi, Stefanie Jegelka +1

We study the statistical-computational trade-offs for learning with exact invariances (or symmetries) using kernel regression. Traditional methods, such as data augmentation, group…

Subsampling Graphs with GNN Performance Guarantees

Mika Sarkin Jain, Stefanie Jegelka, Ishani Karmarkar +2

How can we subsample graph data so that a graph neural network (GNN) trained on the subsample achieves performance comparable to training on the full dataset? This question is of f…

cs.LG2025★ 1 cited

On the Emergence of Position Bias in Transformers

Xinyi Wu, Yifei Wang, Stefanie Jegelka +1

Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive…

Understanding the Role of Equivariance in Self-supervised Learning

Yifei Wang, Kaiwen Hu, Sharut Gupta +3

Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by b…

Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness

Qi Zhang, Yifei Wang, Jingyi Cui +4

Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in uncle…

In-Context Symmetries: Self-Supervised Learning through Contextual World Models

Sharut Gupta, Chenyu Wang, Yifei Wang +2

At the core of self-supervised learning for vision is the idea of learning invariant or equivariant representations with respect to a set of data transformations. This approach, ho…

On the Role of Attention Masks and LayerNorm in Transformers

Xinyi Wu, Amir Ajorlou, Yifei Wang +2

Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers…

cs.LG2024★ 3 cited

A Theoretical Understanding of Self-Correction through In-context Alignment

Yifei Wang, Yuyang Wu, Zeming Wei +2

Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities pure…

A Canonicalization Perspective on Invariant and Equivariant Learning

George Ma, Yifei Wang, Derek Lim +2

In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods em…