From the 2 of 64 linked papers with an AI index.
1 citations · 1 across the 25 of their papers we have counts for
48 papers · 1 filter
The Sparsity Whisperer
Linghao Kong, Inimai Subramanian, Micah Adler +3
Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a…
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2
The paper introduces DarwinLM, an evolutionary search method for training-aware structured pruning of large language models that integrates lightweight post‑pruning training to fin…
DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis +2
Shampoo is one of the leading approximate second-order optimizers: a variant of it has won the MLCommons AlgoPerf competition, and it has been shown to produce models with lower ac…
CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
Soroush Tabesh, Mher Safaryan, Andrei Panferov +2
Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAG…
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
Andrej JovanoviÄ, Alex Iacob, Mher Safaryan +6
Distributed training of foundation models via is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they…
Apertus LLM Family Expansion via Distillation and Quantization
Andrei Panferov, Davit Melikidze, Martin Jaggi +1
The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to s…