12 citations · 32 across the 23 of their papers we have counts for
23 papers
FlashSampling: Fast and Memory-Efficient Exact Sampling
Tomas Ruiz, Zhen Qin, Yifan Zhang +3
Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We…
Elucidating the Design Space of Decay in Linear Attention
Zhen Qin, Xuyang Shen, Yiran Zhong
This paper presents a comprehensive investigation into the decay mechanisms inherent in linear complexity sequence models. We systematically delineate the design space of decay mec…
Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
Yan Ma, Steffi Chern, Xuyang Shen +2
Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-langua…
Multi-modal Time Series Analysis: A Tutorial and Survey
Yushan Jiang, Kanghui Ning, Zijie Pan +7
Multi-modal time series analysis has recently emerged as a prominent research area in data mining, driven by the increasing availability of diverse data modalities, such as text, i…
MiniMax-01: Scaling Foundation Models with Lightning Attention
MiniMax, Aonian Li, Bangwei Gong +87
We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer conte…
Scaling Laws for Linear Complexity Language Models
Xuyang Shen, Dong Li, Ruitao Leng +3
The interest in linear complexity models for large language models is on the rise, although their scaling capacity remains uncertain. In this study, we present the scaling laws for…
Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
Zhen Qin, Weigao Sun, Dong Li +3
We present Lightning Attention, the first linear attention implementation that maintains a constant training speed for various sequence lengths under fixed memory consumption. Due…
You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
Zhen Qin, Yuxin Mao, Xuyang Shen +4
Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism…
Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
Zhen Qin, Xuyang Shen, Dong Li +4
We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention…
TAVGBench: Benchmarking Text to Audible-Video Generation
Yuxin Mao, Xuyang Shen, Jing Zhang +5
The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both a…
HGRN2: Gated Linear RNNs with State Expansion
Zhen Qin, Songlin Yang, Weixuan Sun +4
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However,…
Linear Attention Sequence Parallelism
Weigao Sun, Zhen Qin, Dong Li +3
Sequence parallelism (SP) serves as a prevalent strategy to handle long sequences that exceed the memory limit of a single device. However, for linear sequence modeling methods lik…
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
Weigao Sun, Zhen Qin, Weixuan Sun +5
The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-perf…
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
Zhen Qin, Weigao Sun, Dong Li +3
Linear attention is an efficient attention mechanism that has recently emerged as a promising alternative to conventional softmax attention. With its ability to process tokens in l…
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
Zhen Qin, Dong Li, Weigao Sun +8
We present TransNormerLLM, the first linear attention-based Large Language Model (LLM) that outperforms conventional softmax attention-based models in terms of both accuracy and ef…
Fine-grained Audible Video Description
Xuyang Shen, Dong Li, Jinxing Zhou +9
We explore a new task for audio-visual-language modeling called fine-grained audible video description (FAVD). It aims to provide detailed textual descriptions for the given audibl…
Audio-Visual Segmentation with Semantics
Jinxing Zhou, Xuyang Shen, Jianyuan Wang +8
We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame…
Linear Video Transformer with Feature Fixation
Kaiyue Lu, Zexiang Liu, Jianyuan Wang +8
Vision Transformers have achieved impressive performance in video classification, while suffering from the quadratic complexity caused by the Softmax attention mechanism. Some stud…
AMF: Adaptable Weighting Fusion with Multiple Fine-tuning for Image Classification
Xuyang Shen, Jo Plested, Sabrina Caldwell +2
Fine-tuning is widely applied in image classification tasks as a transfer learning approach. It re-uses the knowledge from a source task to learn and obtain a high performance in t…
Feature Selection on Thermal-stress Dataset
Xuyang Shen, Jo Plested, Tom Gedeon
Physical symptoms caused by high stress commonly happen in our daily lives, leading to the importance of stress recognition systems. This study aims to improve stress classificatio…
Non-binary deep transfer learning for image classification
Jo Plested, Xuyang Shen, Tom Gedeon
The current standard for a variety of computer vision tasks using smaller numbers of labelled training examples is to fine-tune from weights pre-trained on a large image classifica…
Exploring Biases and Prejudice of Facial Synthesis via Semantic Latent Space
Xuyang Shen, Jo Plested, Sabrina Caldwell +1
Deep learning (DL) models are widely used to provide a more convenient and smarter life. However, biased algorithms will negatively influence us. For instance, groups targeted by b…
Pairwise-GAN: Pose-based View Synthesis through Pair-Wise Training
Xuyang Shen, Jo Plested, Yue Yao +1
Three-dimensional face reconstruction is one of the popular applications in computer vision. However, even state-of-the-art models still require frontal face as inputs, which restr…