7 papers
A geometric framework for momentum-based optimizers for low-rank training
Steffen Schotthöfer, Timon Klein, Jonas Kusch
Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank par…
Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
Steffen Schotthöfer, H. Lexie Yang, Stefan Schnake
Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness…
Construction of high-order conservative basis-update and Galerkin dynamical low-rank integrators
Lukas Einkemmer, Jonas Kusch, Steffen Schotthöfer
Numerical simulations of kinetic problems can become prohibitively expensive due to their large memory requirements and computational costs. A method that has proven to successfull…
An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training
Jonas Kusch, Steffen Schotthöfer, Alexandra Walter
Layer factorization has emerged as a widely used technique for training memory-efficient neural networks. However, layer factorization methods face several challenges, particularly…
Windowing Regularization Techniques for Unsteady Aerodynamic Shape Optimization
Steffen Schotthöfer, Beckett Y. Zhou, Tim Albring +1
Unsteady Aerodynamic Shape Optimization presents new challenges in terms of sensitivity analysis of time-dependent objective functions. In this work, we consider periodic unsteady…
GeoLoRA: Geometric integration for parameter efficient fine-tuning
Steffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti +2
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face se…