15.4k citations · 15.8k across the 20 of their papers we have counts for
9 papers · 1 filter
The Key to Going Linear: Analysis-Driven Transformer Linearization
Anna Kuzina, Paul N. Whatmough, Babak Ehteshami Bejnordi
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to ident…
Efficient Reasoning on the Edge
Yelysei Bondarenko, Thomas Hehn, Rob Hesselink +15
Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large…
KaVa: Latent Reasoning via Compressed KV-Cache Distillation
Anna Kuzina, Maciej Pioro, Paul N. Whatmough +1
Large Language Models (LLMs) excel at multi-step reasoning problems with explicit chain-of-thought (CoT), but verbose traces incur significant computational costs and memory overhe…
Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
Leyla Mirvakhabova, Babak Ehteshami Bejnordi, Gaurav Kumar +3
Upcycling pre-trained dense models into sparse Mixture-of-Experts (MoEs) efficiently increases model capacity but often suffers from poor expert specialization due to naive weight…
Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
Andrii Skliar, Ties van Rozendaal, Romain Lepert +5
Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. H…
InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning
Babak Ehteshami Bejnordi, Gaurav Kumar, Amelie Royer +3
Jointly learning multiple tasks with a unified model can improve accuracy and data efficiency, but it faces the challenge of task interference, where optimizing one task objective…
Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
Benjamin Bergner, Andrii Skliar, Amelie Royer +3
Large language models (LLMs) have become ubiquitous in practice and are widely used for generation tasks such as translation, summarization and instruction following. However, thei…
Scalarization for Multi-Task and Multi-Domain Learning at Scale
Amelie Royer, Tijmen Blankevoort, Babak Ehteshami Bejnordi
Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone hence improves model efficien…
Batch-Shaping for Learning Conditional Channel Gated Networks
Babak Ehteshami Bejnordi, Tijmen Blankevoort, Max Welling
We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learnin…