4 papers
Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
Minghui Liu, Aadi Palnitkar, Tahseen Rabbani +9
Large language models (LLMs) have demonstrated remarkable performance on long-context tasks, but are often bottlenecked by memory constraints. Namely, the KV cache, which is used t…
BgGPT 1.0: Extending English-centric LLMs to other languages
Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov +3
We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct: continually pretrained and fine-tuned versions of Google's Gemma-2 models, specifically optimized for Bulgarian…
Mechanistic Design and Scaling of Hybrid Architectures
Michael Poli, Armin W Thomas, Eric Nguyen +9
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale…
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging
Anton Alexandrov, Veselin Raychev, Mark Niklas Müller +3
As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different…