From the 1 of 5 linked papers with an AI index.
5 papers
A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llms
Rahul Krishnan, Volker Schulz
The paper introduces JoLT, a method that compresses the key‑value cache of transformer models by applying a partial Tucker decomposition on token and feature dimensions and adding…
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
Mohammed Adnan, Rohan Jain, Tom Jacobs +4
Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST…
MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models
Chen-Hao Chao, Wei-Fang Sun, Junwei Quan +2
Masked diffusion models (MDM) exhibit superior generalization when learned using a Partial masking scheme (Prime). This approach converts tokens into sub-tokens and models the diff…
Mitigating Privacy Risk via Forget Set-Free Unlearning
Aviraj Newatia, Michael Cooper, Viet Nguyen +1
Training machine learning models requires the storage of large datasets, which often contain sensitive or private data. Storing data is associated with a number of potential risks…
Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
Mohammed Adnan, Rohan Jain, Ekansh Sharma +2
The Lottery Ticket Hypothesis (LTH) suggests there exists a sparse LTH mask and weights that achieve the same generalization performance as the dense model while using significantl…