From the 1 of 6 linked papers with an AI index.
6 papers
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Shengkun Tang, Oliver Sieberling, Eldar Kurtic +2
The paper introduces DarwinLM, an evolutionary search method for training-aware structured pruning of large language models that integrates lightweight post‑pruning training to fin…
Variable-Width Transformers
Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
Dynamic Short Convolutions Improve Transformers
Oliver Sieberling, Bharat Runwal, Rameswar Panda +1
Transformers have become the dominant architecture for large language models, largely due to the scalability and flexibility of attention, feed-forward layers, residual connections…
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Roberto L. Castro, Andrei Panferov, Soroush Tabesh +5
Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those pur…
EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic +1
The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pr…