5 papers
Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli +3
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this tren…
A Convexity-dependent Two-Phase Training Algorithm for Deep Neural Networks
Tomas Hrycej, Bernhard Bermeitinger, Massimo Pavone +2
The key task of machine learning is to minimize the loss function that measures the model fit to the training data. The numerical methods to do this efficiently depend on the prope…
The Structure of Financial Equity Research Reports -- Identification of the Most Frequently Asked Questions in Financial Analyst Reports to Automate Equity Research Using Llama 3 and GPT-4
Adria Pop, Jan Spörer
This research dissects financial equity research reports (ERRs) by mapping their content into categories. There is insufficient empirical analysis of the questions answered in ERRs…
Efficient Neural Network Training via Subset Pretraining
Jan Spörer, Bernhard Bermeitinger, Tomas Hrycej +2
In training neural networks, it is common practice to use partial gradients computed over batches, mostly very small subsets of the training set. This approach is motivated by the…
Reducing the Transformer Architecture to a Minimum
Bernhard Bermeitinger, Tomas Hrycej, Massimo Pavone +2
Transformers are a widespread and successful model architecture, particularly in Natural Language Processing (NLP) and Computer Vision (CV). The essential innovation of this archit…