5 papers
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-AI, :, Anyi Xu +585
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation…
Accelerating Attention with Basis Decomposition
Jialin Zhao
Attention is a core operation in large language models (LLMs). We present BD Attention (BDA), a lossless algorithmic reformulation of attention. BDA is enabled by a simple matrix i…
Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Yingtao Zhang, Diego Cerretti, Jialin Zhao +4
Dynamic sparse training (DST) can reduce the computational demands in ANNs, but faces difficulties in keeping peak performance at high sparsity levels. The Cannistraci-Hebb trainin…
Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
Jialin Zhao, Yingtao Zhang, Carlo Vittorio Cannistraci
The rapid growth of Large Language Models has driven demand for effective model compression techniques to reduce memory and computation costs. Low-rank pruning has gained attention…
Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Jialin Zhao, Yingtao Zhang, Xinghang Li +2
The growing demands on GPU memory posed by the increasing number of neural network parameters call for training approaches that are more memory-efficient. Previous memory reduction…