4 papers
Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
Jinrui Zhang, Chaodong Xiao, Aoqi Wu +2
Pretraining large language models (LLMs) typically requires centralized clusters with thousands of high-memory GPUs (e.g., H100/A100). Recent decentralized training methods reduce…
BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
Chaodong Xiao, Zhengqiang Zhang, Lei Zhang
Transformers have achieved widespread and remarkable success, while the computational complexity of their attention modules remains a major bottleneck for vision tasks. Existing me…
Polyline Path Masked Attention for Vision Transformer
Zhongchen Zhao, Chaodong Xiao, Hui Lin +3
Global dependency modeling and spatial position modeling are two core issues of the foundational architecture design in current deep learning frameworks. Recently, Vision Transform…
Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion
Chaodong Xiao, Minghan Li, Zhengqiang Zhang +2
Selective state space models (SSMs), such as Mamba, highly excel at capturing long-range dependencies in 1D sequential data, while their applications to 2D vision tasks still face…