8 papers
Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
Yu-Hang Wu, Qin-Yuan Liu, Qiu-Yang Zhao +3
Selective layer-wise updates are essential for low-cost continued pre-training of Large Language Models (LLMs), yet determining which layers to freeze or train remains an empirical…
LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection
Haohui Zhang, Zhiye Wang, Xiaoying Gan +2
Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing. The parallel capabilities of existing dLLMs stem from the a…
Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams
Zhenghui Guo, Yuanbin Man, Junyuan Sheng +8
Real-time understanding of long video streams remains challenging for multimodal large language models (VLMs) due to redundant frame processing and rapid forgetting of past context…
Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model
Hongyang Wei, Baixin Xu, Hongbo Liu +18
Recent advances in multimodal models have demonstrated impressive capabilities in unified image generation and editing. However, many prominent open-source models prioritize scalin…
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Huanyu Wang, Jushi Kai, Haoli Bai +4
Vision-Language Models (VLMs) incur substantial computational overhead and inference latency due to the large number of vision tokens introduced by high-resolution image and video…
FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
Jushi Kai, Yixuan Wang, Boyi Zeng +4
Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilli…
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
Jian Yuan, Ziwei He, Haoli Bai +2
Large Language Models (LLMs) use key-value (KV) cache to reduce redundant computation in autoregressive generation. However, the KV cache size increases linearly during generation,…
TreeKV: Smooth Key-Value Cache Compression with Tree Structures
Ziwei He, Jian Yuan, Haoli Bai +2
Efficient key-value (KV) cache compression is critical for scaling transformer-based Large Language Models (LLMs) in long sequences and resource-limited settings. Existing methods…