4 papers
Hunyuan-A13B Technical Report
Tencent Hunyuan Team, Ao Liu, Botong Zhou +72
We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion durin…
Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
Dian Jiao, Jiaxin Duan, Shuai Zhao +3
Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propo…
Global Context Compression with Interleaved Vision-Text Transformation
Dian Jiao, Jiaxin Duan, Shuai Zhao +3
Recent achievements of vision-language models in end-to-end OCR point to a new avenue for low-loss compression of textual information. This motivates earlier works that render the…
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
Tencent Hunyuan Team, Ao Liu, Botong Zhou +248
As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mam…