4 papers
Continuous Latent Diffusion Language Model
Hongcan Guo, Qinyu Zhao, Yian Zhao +8
Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing…
CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
Haoran Li, Sucheng Ren, Alan Yuille +1
Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts,…
VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents
Feng Wang, Yichun Shi, Ceyuan Yang +4
This work presents VTok, a unified video tokenization framework that can be used for both generation and understanding tasks. Unlike the leading vision-language systems that tokeni…
Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane
Haoyu Liu, Sucheng Ren, Tingyu Zhu +5
Rotary Position Embedding (RoPE) is the de facto positional encoding in large language models due to its ability to encode relative positions and support length extrapolation. When…