7 papers
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Xin Cheng, Xingkai Yu, Chenze Shao +30
Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose lo…
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Anyi Xu, Bangcai Lin +315
We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSe…
General learned delegation by clones
Darren Li, Meiqi Chen, Chenze Shao +2
Frontier language models improve with additional test-time computation, but serial reasoning or uncoordinated parallel sampling can be compute-inefficient under fixed inference bud…
Continuous Autoregressive Language Models
Chenze Shao, Darren Li, Fandong Meng +1
The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a…
Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Zhengrui Ma, Yang Feng, Chenze Shao +3
We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregres…
Beyond Next Token Prediction: Patch-Level Training for Large Language Models
Chenze Shao, Fandong Meng, Jie Zhou
The prohibitive training costs of Large Language Models (LLMs) have emerged as a significant bottleneck in the development of next-generation LLMs. In this paper, we show that it i…