42 citations · 214 across the 65 of their papers we have counts for
7 papers · 1 filter
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Yushuai Sun, Zikun Zhou, Lin Gao +2
Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still re…
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Ruijie Zhou, Fanxu Meng, Yufei Xu +4
DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix toke…
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Yufei Xu, Fanxu Meng, Fan Jiang +11
Token-level sparse attention mechanisms, exemplified by DeepSeek Sparse Attention (DSA), achieve fine-grained key selection by scoring every historical key for each query through a…
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Pingzhi Tang, Ruijie Zhou, Fanxu Meng +2
Current quantization methods for LLMs predominantly rely on block-wise structures to maintain efficiency, often at the cost of representational flexibility. In this work, we demons…
Saliency-Aware Regularized Graph Neural Network
Wenjie Pei, Weina Xu, Zongze Wu +4
The crux of graph classification lies in the effective representation learning for the entire graph. Typical graph neural networks focus on modeling the local dependencies when agg…
Label-Aware Distribution Calibration for Long-tailed Classification
Chaozheng Wang, Shuzheng Gao, Cuiyun Gao +4
Real-world data usually present long-tailed distributions. Training on imbalanced data tends to render neural networks perform well on head classes while much worse on tail classes…
Time Series Classification using the Hidden-Unit Logistic Model
Wenjie Pei, Hamdi Dibeklioğlu, David M. J. Tax +1
We present a new model for time series classification, called the hidden-unit logistic model, that uses binary stochastic hidden units to model latent structure in the data. The hi…