25 citations · 31 across the 22 of their papers we have counts for
23 papers
Block-Sparse Attention with Semantic-Geometric Decoupled Routing
Xinwei Long, Weigao Sun, Weibo Gao +6
Long-context inference has become a defining capability of large language models, but exact dense attention remains costly due to its quadratic scaling with sequence length. Block-…
MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
Yi Zhu, Xiongwei Wu, Qiyi Wang +8
Mobile planning requires a central agent to coordinate tools, persistent user context, reusable procedures, and specialized sub-agents while adapting to execution feedback. Evaluat…
Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents
Ming Ma, Yi Zhu, Yiran Zhong +7
Omni-modal agents must seek evidence across video, audio, web pages, and computation to answer questions. Their main bottleneck is planning: noisy multimodal observations accumulat…
FlashSampling: Fast and Memory-Efficient Exact Sampling
Tomas Ruiz, Zhen Qin, Yifan Zhang +3
Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We…
Elucidating the Design Space of Decay in Linear Attention
Zhen Qin, Xuyang Shen, Yiran Zhong
This paper presents a comprehensive investigation into the decay mechanisms inherent in linear complexity sequence models. We systematically delineate the design space of decay mec…
Learning Spatial Decay for Vision Transformers
Yuxin Mao, Zhen Qin, Jinxing Zhou +4
Vision Transformers (ViTs) have revolutionized computer vision, yet their self-attention mechanism lacks explicit spatial inductive biases, leading to suboptimal performance on spa…
Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective
Yuxin Mao, Zhen Qin, Jinxing Zhou +6
Autoregressive (AR) models have garnered significant attention in image generation for their ability to effectively capture both local and global structures within visual data. How…
Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
Yan Ma, Steffi Chern, Xuyang Shen +2
Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-langua…
LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid
Weigao Sun, Disen Lan, Yiran Zhong +2
Linear sequence modeling approaches, such as linear attention, provide advantages like linear-time training and constant-memory inference over sequence lengths. However, existing s…
MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax, Aonian Li, Bangwei Gong +87
We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer conte…
A Generative Victim Model for Segmentation
Aixuan Li, Jing Zhang, Jiawei Shi +2
We find that the well-trained victim models (VMs), against which the attacks are generated, serve as fundamental prerequisites for adversarial attacks, i.e. a segmentation VM is ne…
Deep Non-rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling
Hui Deng, Jiawei Shi, Zhen Qin +2
Non-Rigid Structure-from-Motion (NRSfM) is a classic 3D vision problem, where a 2D sequence is taken as input to estimate the corresponding 3D sequence. Recently, the deep neural n…
Scaling Laws for Linear Complexity Language Models
Xuyang Shen, Dong Li, Ruitao Leng +3
The interest in linear complexity models for large language models is on the rise, although their scaling capacity remains uncertain. In this study, we present the scaling laws for…
Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
Zhen Qin, Weigao Sun, Dong Li +3
We present Lightning Attention, the first linear attention implementation that maintains a constant training speed for various sequence lengths under fixed memory consumption. Due…
You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
Zhen Qin, Yuxin Mao, Xuyang Shen +4
Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism…
Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
Zhen Qin, Xuyang Shen, Dong Li +4
We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention…
TAVGBench: Benchmarking Text to Audible-Video Generation
Yuxin Mao, Xuyang Shen, Jing Zhang +5
The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both a…
HGRN2: Gated Linear RNNs with State Expansion
Zhen Qin, Songlin Yang, Weixuan Sun +4
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However,…
Linear Attention Sequence Parallelism
Weigao Sun, Zhen Qin, Dong Li +3
Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods lik…
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
Weigao Sun, Zhen Qin, Weixuan Sun +5
The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-perf…
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
Zhen Qin, Weigao Sun, Dong Li +3
Linear attention is an efficient attention mechanism that has recently emerged as a promising alternative to conventional softmax attention. With its ability to process tokens in l…
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity
Zhen Qin, Yiran Zhong
Toeplitz Neural Networks (TNNs) have exhibited outstanding performance in various sequence modeling tasks. They outperform commonly used Transformer-based models while benefiting f…
Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Zhen Qin, Songlin Yang, Yiran Zhong
Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest i…