28 citations · 72 across the 11 of their papers we have counts for
11 papers
Efficient Neural Network Training via Subset Pretraining
Jan Spörer, Bernhard Bermeitinger, Tomas Hrycej +2
In training neural networks, it is common practice to use partial gradients computed over batches, mostly very small subsets of the training set. This approach is motivated by the…
Reducing the Transformer Architecture to a Minimum
Bernhard Bermeitinger, Tomas Hrycej, Massimo Pavone +2
Transformers are a widespread and successful model architecture, particularly in Natural Language Processing (NLP) and Computer Vision (CV). The essential innovation of this archit…
Make Deep Networks Shallow Again
Bernhard Bermeitinger, Tomas Hrycej, Siegfried Handschuh
Deep neural networks have a good success record and are thus viewed as the best architecture choice for complex applications. Their main shortcoming has been, for a long time, the…
Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision
Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
Determining an appropriate number of attention heads on one hand and the number of transformer-encoders, on the other hand, is an important choice for Computer Vision (CV) tasks us…
Training Neural Networks in Single vs Double Precision
Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh
The commitment to single-precision floating-point arithmetic is widespread in the deep learning community. To evaluate whether this commitment is justified, the influence of comput…
Multimodal Semantic Transfer from Text to Image. Fine-Grained Image Classification by Distributional Semantics
Simon Donig, Maria Christoforaki, Bernhard Bermeitinger +1
In the last years, image classification processes like neural networks in the area of art-history and Heritage Informatics have experienced a broad distribution (Lang and Ommer 201…