257 citations · 621 across the 31 of their papers we have counts for
7 papers · 1 filter
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Aixin Liu, Bei Feng +154
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which…
Exploring Activation Patterns of Parameters in Language Models
Yudong Wang, Damai Dai, Zhifang Sui
Most work treats large language models as black boxes without in-depth understanding of their internal working mechanism. In order to explain the internal representations of LLMs,…
Large Language Models Struggle with Unreasonability in Math Problems
Jingyuan Ma, Damai Dai, Zihang Yuan +6
Large Language Models (LLMs) have shown remarkable success on a wide range of math and reasoning benchmarks. However, we observe that they often struggle when faced with unreasonab…
PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
Xiangdi Meng, Damai Dai, Weiyao Luo +7
Supervised fine-tuning is the most common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning LLMs requires massive computational resources. Rece…
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Damai Dai, Chengqi Deng, Chenggang Zhao +14
In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional M…
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Xiao Bi +85
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions,…
Language Models Encode the Value of Numbers Linearly
Fangwei Zhu, Damai Dai, Zhifang Sui
Large language models (LLMs) have exhibited impressive competence in various tasks, but their internal mechanisms on mathematical problems are still under-explored. In this paper,…