257 citations · 621 across the 31 of their papers we have counts for
6 papers · 2 filters
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Aixin Liu, Bei Feng +154
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which…
Large Language Models Struggle with Unreasonability in Math Problems
Jingyuan Ma, Damai Dai, Zihang Yuan +6
Large Language Models (LLMs) have shown remarkable success on a wide range of math and reasoning benchmarks. However, we observe that they often struggle when faced with unreasonab…
PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
Xiangdi Meng, Damai Dai, Weiyao Luo +7
Supervised fine-tuning is the most common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning LLMs requires massive computational resources. Rece…
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Damai Dai, Chengqi Deng, Chenggang Zhao +14
In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional M…
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Xiao Bi +85
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions,…
Language Models Encode the Value of Numbers Linearly
Fangwei Zhu, Damai Dai, Zhifang Sui
Large language models (LLMs) have exhibited impressive competence in various tasks, but their internal mechanisms on mathematical problems are still under-explored. In this paper,…