6 papers
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…
Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
Chuanyang Zheng, Jiankai Sun, Yihang Gao +13
Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs). Traditionally, MoE relies on as the router score funct…
SAS: Simulated Attention Score
Chuanyang Zheng, Jiankai Sun, Yihang Gao +12
The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA),…
ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation
Suning Huang, Qianzhong Chen, Xiaohan Zhang +2
3D world models (i.e., learning-based 3D dynamics models) offer a promising approach to generalizable robotic manipulation by capturing the underlying physics of environment evolut…
Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation
Chengyang He, Gadiel Sznaier Camps, Xu Liu +2
We present Latent Theory of Mind (LatentToM), a decentralized diffusion policy architecture for collaborative robot manipulation. Our policy allows multiple manipulators with their…
Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives
Chengyang He, Xu Liu, Gadiel Sznaier Camps +2
Diffusion policies have demonstrated remarkable dexterity and robustness in intricate, high-dimensional robot manipulation tasks, while training from a small number of demonstratio…