9 papers
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
Wei He, Xianghan Meng, Zhiyuan Huang +3
Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set reco…
Open-Text Aerial Detection: A Unified Framework For Aerial Visual Grounding And Detection
Guoting Wei, Xia Yuan, Yang Zhou +6
Open-Vocabulary Aerial Detection (OVAD) and Remote Sensing Visual Grounding (RSVG) have emerged as two key paradigms for aerial scene understanding. However, each paradigm suffers…
Delving into Muon and Beyond: Deep Analysis and Extensions
Xianbiao Qi, Marco Chen, Jiaquan Ye +2
The Muon optimizer has recently attracted considerable attention for its strong empirical performance and use of orthogonalized updates on matrix-shaped parameters, yet its underly…
SimpleGPT: Improving GPT via A Simple Normalization Strategy
Marco Chen, Xianbiao Qi, Yelin He +2
In this work, we revisit Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the He…
HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
Dongquan Yang, Yifan Yang, Xiaotian Yu +2
Processing long-context inputs with large language models presents a significant challenge due to the enormous memory requirements of the Key-Value (KV) cache during inference. Exi…
DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
Xianbiao Qi, Marco Chen, Wenjie Xiao +4
Transformers have become the de facto backbone of modern deep learning, yet their training typically demands an advanced optimizer with adaptive learning rate like AdamW, rather th…