9 citations · 26 across the 20 of their papers we have counts for
20 papers
DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Models
Zirui Wu, Lin Zheng, Jiacheng Ye +5
Block diffusion language models accelerate decoding through parallel block-wise denoising, yet whether they can be reliably scaled for long chain-of-thought (CoT) reasoning remains…
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3
Tokenizer-free language models eliminate the tokenizer step of the language modeling pipeline by operating directly on bytes; patch-based variants further aggregate contiguous byte…
Proxy Compression for Language Modeling
Lin Zheng, Xinyu Li, Qian Liu +2
Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby cou…
DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
Zirui Wu, Lin Zheng, Zhihui Xie +8
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, th…
Dream-Coder 7B: An Open Diffusion Language Model for Code
Zhihui Xie, Jiacheng Ye, Lin Zheng +8
We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoreg…
Dream 7B: Diffusion Large Language Models
Jiacheng Ye, Zhihui Xie, Lin Zheng +5
We introduce Dream 7B, the most powerful open diffusion large language model to date. Unlike autoregressive (AR) models that generate tokens sequentially, Dream 7B employs discrete…
Scaling Diffusion Language Models via Adaptation from Autoregressive Models
Shansan Gong, Shivam Agarwal, Yizhe Zhang +9
Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, c…
SubgoalXL: Subgoal-based Expert Learning for Theorem Proving
Xueliang Zhao, Lin Zheng, Haige Bo +3
Formal theorem proving, a field at the intersection of mathematics and computer science, has seen renewed interest with advancements in large language models (LLMs). This paper int…
Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models
Jiacheng Ye, Shansan Gong, Liheng Chen +8
Recently, diffusion models have garnered significant interest in the field of text processing due to their many potential advantages compared to conventional autoregressive models.…
Linear Attention via Orthogonal Memory
Jun Zhang, Shuyang Jiang, Jiangtao Feng +2
Efficient attentions have greatly improved the computational efficiency of Transformers. However, most existing linear attention mechanisms suffer from an \emph{efficiency degradat…
Self-Infilling Code Generation
Lin Zheng, Jianbo Yuan, Zhi Zhang +2
This work introduces self-infilling code generation, a general framework that incorporates infilling operations into auto-regressive decoding. Our approach capitalizes on the obser…
Attentive Multi-Layer Perceptron for Non-autoregressive Generation
Shuyang Jiang, Jun Zhang, Jiangtao Feng +2
Autoregressive~(AR) generation almost dominates sequence generation for its efficacy. Recently, non-autoregressive~(NAR) generation gains increasing popularity for its efficiency a…
Retrieved Sequence Augmentation for Protein Representation Learning
Chang Ma, Haiteng Zhao, Lin Zheng +7
Protein language models have excelled in a variety of tasks, ranging from structure prediction to protein engineering. However, proteins are highly diverse in functions and structu…
Efficient Attention via Control Variates
Lin Zheng, Jianbo Yuan, Chong Wang +1
Random-feature-based attention (RFA) is an efficient approximation of softmax attention with linear runtime and space complexity. However, the approximation gap between RFA and con…
A Reparameterized Discrete Diffusion Model for Text Generation
Lin Zheng, Jianbo Yuan, Lei Yu +1
This work studies discrete diffusion probabilistic models with applications to natural language generation. We derive an alternative yet equivalent formulation of the sampling from…
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
Jun Zhang, Shuyang Jiang, Jiangtao Feng +2
Transformer has achieved remarkable success in language, image, and speech processing. Recently, various efficient attention architectures have been proposed to improve transformer…
Linear Complexity Randomized Self-attention Mechanism
Lin Zheng, Chong Wang, Lingpeng Kong
Recently, random feature attentions (RFAs) are proposed to approximate the softmax attention in linear time and space complexity by linearizing the exponential kernel. In this pape…
Ripple Attention for Visual Perception with Sub-quadratic Complexity
Lin Zheng, Huijie Pan, Lingpeng Kong
Transformer architectures are now central to sequence modeling tasks. At its heart is the attention mechanism, which enables effective modeling of long-term dependencies in a seque…
Cascaded Head-colliding Attention
Lin Zheng, Zhiyong Wu, Lingpeng Kong
Transformers have advanced the field of natural language processing (NLP) on a variety of important tasks. At the cornerstone of the Transformer architecture is the multi-head atte…
Generative Semantic Hashing Enhanced via Boltzmann Machines
Lin Zheng, Qinliang Su, Dinghan Shen +1
Generative semantic hashing is a promising technique for large-scale information retrieval thanks to its fast retrieval speed and small memory footprint. For the tractability of tr…