4 citations · 4 across the 5 of their papers we have counts for
6 papers
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang +4
Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods su…
Tensor Cookbook: Mastering Tensors through Diagrams
Beheshteh T. Rakhshan, Guillaume Rabusseau
High-dimensional data arise naturally in many areas of science and engineering, including machine learning, signal processing, computational physics, and statistics. Such data are…
KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
Damien Lesens, Beheshteh T. Rakhshan, Guillaume Rabusseau
The Key-Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequen…
Efficient Leverage Score Sampling for Tensor Train Decomposition
Vivek Bharadwaj, Beheshteh T. Rakhshan, Osman Asif Malik +1
Tensor Train~(TT) decomposition is widely used in the machine learning and quantum physics communities as a popular tool to efficiently compress high-dimensional tensor data. In th…
Rademacher Random Projections with Tensor Networks
Beheshteh T. Rakhshan, Guillaume Rabusseau
Random projection (RP) have recently emerged as popular techniques in the machine learning community for their ability in reducing the dimension of very high-dimensional tensors. F…
Tensorized Random Projections
Beheshteh T. Rakhshan, Guillaume Rabusseau
We introduce a novel random projection technique for efficiently reducing the dimension of very high-dimensional tensors. Building upon classical results on Gaussian random project…