7 papers
Spark Transformer: Reactivating Sparsity in FFN and Attention
Chong You, Kan Wu, Zhipeng Jia +16
The discovery of the lazy neuron phenomenon in trained Transformers, where the vast majority of neurons in their feed-forward networks (FFN) are inactive for each token, has spurre…
Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression
Dingcheng Zhen, Qian Qiao, Xu Zheng +6
We introduce TransDiff, the first image generation model that marries Autoregressive (AR) Transformer with diffusion models. In this joint modeling framework, TransDiff encodes lab…
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
Tencent Hunyuan Team, Ao Liu, Botong Zhou +248
As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mam…
Scaling Laws for Floating Point Quantization Training
Xingwu Sun, Shuaipeng Li, Ruobing Xie +13
Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer qu…
Baichuan-M1: Pushing the Medical Capability of Large Language Models
Bingning Wang, Haizhou Zhao, Huozhi Zhou +39
The current generation of large language models (LLMs) is typically designed for broad, general-purpose applications, while domain-specific LLMs, especially in vertical fields like…
Accelerating Sparse Graph Neural Networks with Tensor Core Optimization
Ka Wai Wu
Graph neural networks (GNNs) have seen extensive application in domains such as social networks, bioinformatics, and recommendation systems. However, the irregularity and sparsity…