Stefanie Jegelka

How Do Transformers Learn to Represent Symmetries?

Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1

Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning. Despite its empirical success, the…

Training-Free Transformer Merging via Sequential Local Operator Alignment

Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2

Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging…

Efficient Learning and Symmetry Discovery under Exact Invariances

Ashkan Soleymani, Behrooz Tahmasebi, Patrick Jaillet +1

Learning with group invariances is central to many scientific and geometric learning problems, yet its computational foundations remain poorly understood. Even for classical superv…

Data Augmentation: A Fourier Analysis Perspective

Behrooz Tahmasebi, Melanie Weber, Stefanie Jegelka

Data augmentation is a simple and model-agnostic approach for exploiting known invariances in learning problems. Given a group acting on the input space, one augments the training…

cs.IR2026

No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

Lixuan Guo, Yifei Wang, Tiansheng Wen +3

Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-level interactions. However, thi…

Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2

Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL…

Scaling Attention via Feature Sparsity

Yan Xie, Tiansheng Wen, Tangda Huang +4

Scaling Transformers to ultra-long contexts is bottlenecked by the $O (n^{2} d)$ cost of self-attention. Existing methods reduce this cost along the sequence axis through local window…

CSRv2: Unlocking Ultra-Sparse Embeddings

Lixuan Guo, Yifei Wang, Tiansheng Wen +5

In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. Yet widely used dens…

Route Experts by Sequence, not by Token

Tiansheng Wen, Yifei Wang, Aosong Feng +7

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed num…

LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs

Ang Li, Yifei Wang, Zhihang Yuan +2

Reinforcement learning in large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts, forcing the model…

Geometric Algorithms for Neural Combinatorial Optimization with Constraints

Nikolaos Karalias, Akbar Rafiey, Yifei Xu +4

Self-Supervised Learning (SSL) for Combinatorial Optimization (CO) is an emerging paradigm for solving combinatorial problems using neural networks. In this paper, we address a cen…

cs.CV2025

SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning

Xiaojun Guo, Runyu Zhou, Yifei Wang +8

Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequatel…

LieAugmenter: Equivariant Learning by Discovering Symmetries with Learnable Augmentations

Eduardo Santos-Escriche, Ya-Wei Eileen Lin, Stefanie Jegelka

Data augmentation is a powerful mechanism in equivariant machine learning, encouraging symmetry by training networks to produce consistent outputs under transformed inputs. Yet, ef…

G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning

Xiaojun Guo, Ang Li, Yifei Wang +2

Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly ge…

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

Tiansheng Wen, Yifei Wang, Zequn Zeng +7

Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learn…

Learning with Exact Invariances in Polynomial Time

Ashkan Soleymani, Behrooz Tahmasebi, Stefanie Jegelka +1

We study the statistical-computational trade-offs for learning with exact invariances (or symmetries) using kernel regression. Traditional methods, such as data augmentation, group…

Subsampling Graphs with GNN Performance Guarantees

Mika Sarkin Jain, Stefanie Jegelka, Ishani Karmarkar +2

How can we subsample graph data so that a graph neural network (GNN) trained on the subsample achieves performance comparable to training on the full dataset? This question is of f…

cs.AI2025★ 4 cited

When More is Less: Understanding Chain-of-Thought Length in LLMs

Yuyang Wu, Yifei Wang, Ziyu Ye +3

Large Language Models (LLMs) employ Chain-of-Thought (CoT) reasoning to deconstruct complex problems. While longer CoTs are often presumed superior, this paper challenges that noti…

cs.LG2025★ 1 cited

On the Emergence of Position Bias in Transformers

Xinyi Wu, Yifei Wang, Stefanie Jegelka +1

Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive…

Understanding the Role of Equivariance in Self-supervised Learning

Yifei Wang, Kaiwen Hu, Sharut Gupta +3

Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by b…

Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness

Qi Zhang, Yifei Wang, Jingyi Cui +4

Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in uncle…

cs.CL2024★ 2 cited

What is Wrong with Perplexity for Long-context Language Modeling?

Lizhe Fang, Yifei Wang, Zhaoyang Liu +5

Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While r…

In-Context Symmetries: Self-Supervised Learning through Contextual World Models

Sharut Gupta, Chenyu Wang, Yifei Wang +2

At the core of self-supervised learning for vision is the idea of learning invariant or equivariant representations with respect to a set of data transformations. This approach, ho…