12 papers
On the Residual Scaling of Looped Transformers: Stability and Transferability
Shaowen Wang, Bingrui Li, Ge Zhang +3
Looped (weight-tied) Transformers apply a shared residual block times (, same at each step), increasing effective depth without adding p…
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Shaowen Wang, Yuke Zheng, Tansheng Zhu +4
Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling a…
Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation
Jiashuo Sun, Jimeng Shi, Yixuan Xie +10
Retrieval-Augmented Generation (RAG) has become a standard approach for knowledge-intensive question answering, but existing systems remain brittle on multi-hop questions, where so…
ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Neha Verma, Nikhil Mehta, Shao-Chuan Wang +7
Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, languag…
SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection
Kexian Tang, Jiani Wang, Shaowen Wang +1
While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive…
Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space
Xingwei Qu, Shaowen Wang, Zihao Huang +16
Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity…