At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference
arXiv:2607.25504
The paper introduces Ventaglio, a hardware extension and ISA support for vector processors that efficiently executes sparse tensor contractions in Transformer inference, achieving several‑fold speedups over dense and baseline sparse implementations.
Abstract
Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by over optimized RVV baselines, with only area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical dual sparsity, Ventaglio achieves and speedup over dense baselines during prefill and autoregressive decoding, respectively.
5 pages, 4 figures, 34th IFIP/IEEE International Conference on Very Large Scale Integration SoC (VLSI-SoC 2026)