3 citations · 7 across the 19 of their papers we have counts for
19 papers
A Theoretical Framework for Masked Pretraining (MPT)
Qi Zhang, Runyu Zhou, Yifei Wang +1
Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkab…
Beyond the Next Step: Variable-Length Latent World Models for Long-Horizon Planning
Tianqi Du, Qi Zhang, Yifei Wang +1
Recently, world models have emerged as a promising paradigm for building intelligent agents by learning predictive models that estimate future environment states conditioned on obs…
No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval
Lixuan Guo, Yifei Wang, Tiansheng Wen +3
Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-level interactions. However, thi…
Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
Chiyue Wei, Cong Guo, Bowen Duan +7
Large language model (LLM) serving is fundamentally limited by inefficient hardware utilization. Autoregressive (AR) decoding underutilizes GPUs due to its strictly sequential exec…
Scaling Attention via Feature Sparsity
Yan Xie, Tiansheng Wen, Tangda Huang +4
Scaling Transformers to ultra-long contexts is bottlenecked by the cost of self-attention. Existing methods reduce this cost along the sequence axis through local window…
StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding
Guowei Tang, Tianwen Qian, Huanran Zheng +2
Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on…
LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
Ang Li, Yifei Wang, Zhihang Yuan +2
Reinforcement learning in large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts, forcing the model…
SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
Xiaojun Guo, Runyu Zhou, Yifei Wang +8
Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequatel…
G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
Xiaojun Guo, Ang Li, Yifei Wang +2
Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly ge…
On the Emergence of Position Bias in Transformers
Xinyi Wu, Yifei Wang, Stefanie Jegelka +1
Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive…
Understanding the Role of Equivariance in Self-supervised Learning
Yifei Wang, Kaiwen Hu, Sharut Gupta +3
Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by b…
Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness
Qi Zhang, Yifei Wang, Jingyi Cui +4
Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in uncle…
What is Wrong with Perplexity for Long-context Language Modeling?
Lizhe Fang, Yifei Wang, Zhaoyang Liu +5
Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While r…
In-Context Symmetries: Self-Supervised Learning through Contextual World Models
Sharut Gupta, Chenyu Wang, Yifei Wang +2
At the core of self-supervised learning for vision is the idea of learning invariant or equivariant representations with respect to a set of data transformations. This approach, ho…
On the Role of Attention Masks and LayerNorm in Transformers
Xinyi Wu, Amir Ajorlou, Yifei Wang +2
Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers…
A Theoretical Understanding of Self-Correction through In-context Alignment
Yifei Wang, Yuyang Wu, Zeming Wei +2
Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities pure…
A Canonicalization Perspective on Invariant and Equivariant Learning
George Ma, Yifei Wang, Derek Lim +2
In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods em…
Non-negative Contrastive Learning
Yifei Wang, Qi Zhang, Yaoyu Guo +1
Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant…
Identifiable Contrastive Learning with Automatic Feature Importance Discovery
Qi Zhang, Yifei Wang, Yisen Wang
Existing contrastive learning methods rely on pairwise sample contrast to learn data representations, but the learned features often lack clear interpretability f…