5 papers
SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning
Guangyuan Wang, Mads Toftrup, Sebastian Loeschcke +2
Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve local conditioning but require…
M+Adam: Low-Precision Training via Additive-Multiplicative Optimization
Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1
Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precisio…
TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training
Sebastian Loeschcke, David Pitt, Robert Joseph George +5
Scientific problems require resolving multi-scale phenomena across different resolutions and learning solution operators in infinite-dimensional function spaces. Neural operators p…
Coarse-To-Fine Tensor Trains for Compact Visual Representations
Sebastian Loeschcke, Dan Wang, Christian Leth-Espensen +3
The ability to learn compact, high-quality, and easy-to-optimize representations for visual data is paramount to many applications such as novel view synthesis and 3D reconstructio…
LoQT: Low-Rank Adapters for Quantized Pretraining
Sebastian Loeschcke, Mads Toftrup, Michael J. Kastoryano +2
Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training…