1 citations · 1 across the 11 of their papers we have counts for
12 papers
Turnslide: Scalable Multi-Turn Data Synthesis by Walking a Finite-State Machine
Aaron Fainman, Gabriela Kadlecová, Maciej Gryka +7
Small language models are inexpensive to serve and can run on private infrastructure, but base models are often not good enough at multi-turn tool calling, and fine-tuning them nee…
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…
Amortizing Scaling Law Construction Costs
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and paramete…
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates…
An Open-Source Training Dataset for Foundation Models for Black-box Optimization
Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien +1
Most black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to generalize across different optimization domains. Foundation models for…
When is Warmstarting Effective for Scaling Language Models?
Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4
Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited prac…
Evolutionary Architecture Search through Grammar-Based Sequence Alignment
Adri Gómez Martín, Felix Möller, Steven McDonagh +5
Neural architecture search (NAS) in expressive search spaces is a computationally hard problem, but it also holds the potential to automatically discover completely novel and perfo…
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik +5
Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introd…
Hyperband-based Bayesian Optimization for Black-box Prompt Selection
Lennart Schneider, Martin Wistuba, Aaron Klein +3
Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks, especially in black-box settings where models are only accessible via…
Warmstarting for Scaling Language Models
Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4
Scaling model sizes to scale performance has worked remarkably well for the current large language models paradigm. The research and empirical findings of various scaling studies l…
Compressing Large Language Models with Automated Sub-Network Search
Rhea Sanjay Sukthanker, Benedikt Staffler, Frank Hutter +1
Large Language Models (LLMs) demonstrate exceptional reasoning abilities, enabling strong generalization across diverse tasks such as commonsense reasoning and instruction followin…
Structural Pruning of Pre-trained Language Models via Neural Architecture Search
Aaron Klein, Jacek Golebiowski, Xingchen Ma +2
Pre-trained language models (PLM), for example BERT or RoBERTa, mark the state-of-the-art for natural language understanding task when fine-tuned on labeled data. However, their la…