11 citations · 13 across the 6 of their papers we have counts for
6 papers
Long-Short Alignment for Effective Long-Context Modeling in LLMs
Tianqi Du, Haotian Huang, Yifei Wang +1
Large language models (LLMs) have exhibited impressive performance and surprising emergent properties. However, their effectiveness remains limited by the fixed context window of t…
Advancing LLM Safe Alignment with Safety Representation Ranking
Tianqi Du, Zeming Wei, Quan Chen +2
The rapid advancement of large language models (LLMs) has demonstrated milestone success in a variety of tasks, yet their potential for generating harmful content has raised signif…
On the Role of Discrete Tokenization in Visual Representation Learning
Tianqi Du, Yifei Wang, Yisen Wang
In the realm of self-supervised learning (SSL), masked image modeling (MIM) has gained popularity alongside contrastive learning methods. MIM involves reconstructing masked regions…
Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
Qi Zhang, Tianqi Du, Haotian Huang +2
In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the va…
ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond
Xiaojun Guo, Yifei Wang, Tianqi Du +1
Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases. Instead of chara…
A Message Passing Perspective on Learning Dynamics of Contrastive Learning
Yifei Wang, Qi Zhang, Tianqi Du +3
In recent years, contrastive learning achieves impressive results on self-supervised visual representation learning, but there still lacks a rigorous understanding of its learning…