4 papers
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Shaowen Wang, Yuke Zheng, Tansheng Zhu +4
Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling a…
Revealing the Learning Dynamics of Long-Context Continual Pre-training
Yupu Liang, Shuang Chen, Guanwei Zhang +2
Existing studies on Long-Context Continual Pre-training (LCCP) mainly focus on small-scale models and limited data regimes (tens of billions of tokens). We argue that directly migr…
Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts
Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao +7
Evaluating the ability of large language models (LLMs) to process lengthy contexts is critical, especially for retrieving query-relevant information embedded within them. We introd…
CAdam: Confidence-Based Optimization for Online Learning
Shaowen Wang, Anan Liu, Jian Xiao +9
Modern recommendation systems frequently employ online learning to dynamically update their models with freshly collected data. The most commonly used optimizer for updating neural…