22 citations · 32 across the 27 of their papers we have counts for
29 papers
CoViT: Instance-Correspondence Contrastive Learning for Vision Transformer
Yisen Wang, Zhirong Wu, Limin Wang
Vision Transformers (ViT) excel in semantic understanding but fail to discriminate between object instances (e.g., identical embeddings for two dogs), limiting their use in instanc…
Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning
Lizhe Fang, Weizhou Shen, Tianyi Tang +1
Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an i…
BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification
Yi-Siang Wang, Kuan-Yu Chen, Yu-Chen Den +1
Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes re…
SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training
Qi Zhang, Yifei Wang, Xiaohan Wang +4
In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre-training, their effectiven…
Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents
Qirui Mi, Zhijian Ma, Mengyue Yang +4
LLM-driven agents excel at sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse…
Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
Tianqi Du, Lizhe Fang, Weijie Yang +4
Diffusion language models enable any-order generation and bidirectional conditioning, offering appealing flexibility for tasks such as infilling, rewriting, and self-correction. Ho…
Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGD
Ze Peng, Jian Zhang, Yisen Wang +3
Information-theoretic (IT) generalization bounds have been used to study the generalization of learning algorithms. These bounds are intrinsically data- and algorithm-dependent so…
Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention
Taiye Chen, Zihan Ding, Anjian Li +4
Recent advancements in video generation has shifted from bidirectional models for short videos to autoregressive ones for ultra long video generation. Previous models, which usuall…
LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
Ang Li, Yifei Wang, Zhihang Yuan +2
Reinforcement learning in large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts, forcing the model…
SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
Xiaojun Guo, Runyu Zhou, Yifei Wang +8
Vision-language models (VLMs) have shown remarkable abilities by integrating large language models with visual inputs. However, they often fail to utilize visual evidence adequatel…
Long-Short Alignment for Effective Long-Context Modeling in LLMs
Tianqi Du, Haotian Huang, Yifei Wang +1
Large language models (LLMs) have exhibited impressive performance and surprising emergent properties. However, their effectiveness remains limited by the fixed context window of t…
Rethinking Invariance in In-context Learning
Lizhe Fang, Yifei Wang, Khashayar Gatmiry +2
In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examp…
R^3-VQA: "Read the Room" by Video Social Reasoning
Lixing Niu, Jiapeng Li, Xingping Yu +6
"Read the room" is a significant social reasoning capability in human daily life. Humans can infer others' mental states from subtle social cues. Previous social reasoning tasks an…
G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
Xiaojun Guo, Ang Li, Yifei Wang +2
Although Large Language Models (LLMs) have demonstrated remarkable progress, their proficiency in graph-related tasks remains notably limited, hindering the development of truly ge…
X-Guard: Multilingual Guard Agent for Content Moderation
Bibek Upadhayay, Vahid Behzadan, Ph. D
Large Language Models (LLMs) have rapidly become integral to numerous applications in critical domains where reliability is paramount. Despite significant advances in safety framew…
Understanding the Role of Equivariance in Self-supervised Learning
Yifei Wang, Kaiwen Hu, Sharut Gupta +3
Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by b…
Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness
Qi Zhang, Yifei Wang, Jingyi Cui +4
Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in uncle…
AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation
Zijun Wang, Haoqin Tu, Jieru Mei +3
This paper studies the vulnerabilities of transformer-based Large Language Models (LLMs) to jailbreaking attacks, focusing specifically on the optimization-based Greedy Coordinate…
What is Wrong with Perplexity for Long-context Language Modeling?
Lizhe Fang, Yifei Wang, Zhaoyang Liu +5
Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While r…
Can In-context Learning Really Generalize to Out-of-distribution Tasks?
Qixun Wang, Yifei Wang, Yisen Wang +1
In this work, we explore the mechanism of in-context learning (ICL) on out-of-distribution (OOD) tasks that were not encountered during training. To achieve this, we conduct synthe…
CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generation
Yisen Wang, Yao Teng, Limin Wang
Recognition and generation are two fundamental tasks in computer vision, which are often investigated separately in the exiting literature. However, these two tasks are highly corr…
A Theoretical Understanding of Self-Correction through In-context Alignment
Yifei Wang, Yuyang Wu, Zeming Wei +2
Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities pure…
A Canonicalization Perspective on Invariant and Equivariant Learning
George Ma, Yifei Wang, Derek Lim +2
In many applications, we desire neural networks to exhibit invariance or equivariance to certain groups due to symmetries inherent in the data. Recently, frame-averaging methods em…
How to Craft Backdoors with Unlabeled Data Alone?
Yifei Wang, Wenhan Ma, Stefanie Jegelka +1
Relying only on unlabeled data, Self-supervised learning (SSL) can learn rich features in an economical and scalable way. As the drive-horse for building foundation models, SSL has…