12 papers
Stateful Token Reduction for Long-Video Hybrid VLMs
Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7
Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning. We study token reducti…
Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
Yitong Jiang, Hongjun Wang, Collin McCarthy +15
Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic al…
MKG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
Hyeongcheol Park, Jiyoung Seo, Jaewon Mun +6
Retrieval-Augmented Generation (RAG) has recently been extended to multimodal settings, connecting multimodal large language models (MLLMs) with vast corpora of external knowledge…
Demystifying Data-Driven Probabilistic Medium-Range Weather Forecasting
Jean Kossaifi, Nikola Kovachki, Morteza Mardani +15
The recent revolution in data-driven methods for weather forecasting has lead to a fragmented landscape of complex, bespoke architectures and training strategies, obscuring the fun…
GSPN-2: Efficient Parallel Sequence Modeling
Hongjun Wang, Yitong Jiang, Collin McCarthy +12
Efficient vision transformer remains a bottleneck for high-resolution images and long-video related real-world applications. Generalized Spatial Propagation Network (GSPN) addresse…
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Yonggan Fu, Xin Dong, Shizhe Diao +12
Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has pri…