5 papers
Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
Amal Abed, Ivan Lukic, Jörg K. H. Franke +1
Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and…
Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
Urs Spiegelhalter, Jörg K. H. Franke, Frank Hutter
Adapting language models to new tasks through continued pretraining faces a fundamental trade-off: models must learn new capabilities while avoiding catastrophic forgetting of exis…
Towards Scaling Laws for Symbolic Regression
David Otte, Jörg K. H. Franke, Arbër Zela +2
Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for produci…
Learning in Compact Spaces with Approximately Normalized Transformer
Jörg K. H. Franke, Urs Spiegelhalter, Marianna Nezhurina +3
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the re…
Transfer Learning for Finetuning Large Language Models
Tobias Strangmann, Lennart Purucker, Jörg K. H. Franke +3
As the landscape of large language models expands, efficiently finetuning for specific tasks becomes increasingly crucial. At the same time, the landscape of parameter-efficient fi…