4 citations · 13 across the 16 of their papers we have counts for
13 papers · 1 filter
Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment
David Otte, Jörg K. H. Franke, Frank Hutter
Symbolic regression is the problem of finding an expression that fits a dataset. Solving it requires searching a combinatorial space of candidate expressions. Neural symbolic regre…
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates…
Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
Amal Abed, Ivan Lukic, Jörg K. H. Franke +1
Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and…
Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
Urs Spiegelhalter, Jörg K. H. Franke, Frank Hutter
Adapting language models to new tasks through continued pretraining faces a fundamental trade-off: models must learn new capabilities while avoiding catastrophic forgetting of exis…
Towards Scaling Laws for Symbolic Regression
David Otte, Jörg K. H. Franke, Arbër Zela +2
Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for produci…
Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Marianna Nezhurina, Harsh Raj, Jörg Franke +7
Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproduc…
Learning in Compact Spaces with Approximately Normalized Transformer
Jörg K. H. Franke, Urs Spiegelhalter, Marianna Nezhurina +3
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the re…
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela +3
Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences. However, both Tran…
Fast Optimizer Benchmark
Simon Blauth, Tobias Bürger, Zacharias Häringer +2
In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from mu…
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +4
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consu…
Improving Deep Learning Optimization through Constrained Parameter Regularization
Jörg K. H. Franke, Michael Hefenbrock, Gregor Koehler +1
Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This…
Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Jörg K. H. Franke, Frederic Runge, Frank Hutter
Our world is ambiguous and this is reflected in the data we use to train our algorithms. This is particularly true when we try to model natural processes where collected data is af…
Practitioner Motives to Use Different Hyperparameter Optimization Methods
Niclas Kannengießer, Niklas Hasebrook, Felix Morsbach +5
Programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization and evolutionary algorithms, are highly sample-efficient in identifying optimal hyperparameter…