3 papers
cs.CL2026
Variable-Width Transformers
Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…
cs.CL2025
Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Yanhong Li, Songlin Yang, Shawn Tan +4
Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improv…
cs.LG2025
TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments
Zhangchen Xu, Adriana Meza Soria, Shawn Tan +4
Large Language Model (LLM) agents are rapidly emerging as powerful systems for automating tasks across domains. Yet progress in the open-source community is constrained by the lack…