57 citations · 148 across the 15 of their papers we have counts for
5 papers · 1 filter
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles…
Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
Konstantin Riedl, Konstantinos Spiliopoulos, Justin Sirignano
A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infi…
Global Convergence of Adjoint-Optimized Neural PDEs
Konstantin Riedl, Justin Sirignano, Konstantinos Spiliopoulos
Many engineering and scientific fields have recently become interested in modeling terms in partial differential equations (PDEs) with neural networks, which requires solving the i…
Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
Nicolás García Trillos, Aditya Kumar Akash, Sixu Li +2
Adversarial attacks pose significant challenges in many machine learning applications, particularly in the setting of distributed training and federated learning, where malicious a…
Gradient is All You Need? How Consensus-Based Optimization can be Interpreted as a Stochastic Relaxation of Gradient Descent
Konstantin Riedl, Timo Klock, Carina Geldhauser +1
In this paper, we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO),…