64 citations · 67 across the 12 of their papers we have counts for
12 papers
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Andrey Veprikov, Arman Bolatov, Aleksandr Bogdanov +4
Optimization lies at the core of modern deep learning, yet existing methods often face a fundamental trade-off between adapting to problem geometry and leveraging curvature utiliza…
Loss-Transformation Invariance in the Damped Newton Method
Alexander Shestakov, Sushil Bohara, Samuel Horváth +2
The Newton method is a powerful optimization algorithm, valued for its rapid local convergence and elegant geometric properties. However, its theoretical guarantees are usually lim…
Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees
Artem Agafonov, Vladislav Ryspayev, Samuel Horváth +3
Quasi-Newton methods are widely used for solving convex optimization problems due to their ease of implementation, practical efficiency, and strong local convergence guarantees. Ho…
Polyak Stepsize: Estimating Optimal Functional Values Without Parameters or Prior Knowledge
Farshed Abdukhakimov, Cuong Anh Pham, Samuel Horváth +2
The Polyak stepsize for Gradient Descent is known for its fast convergence but requires prior knowledge of the optimal functional value, which is often unavailable in practice. In…
DAG: Projected Stochastic Approximation Iteration for DAG Structure Learning
Klea Ziu, Slavomír Hanzely, Loka Li +3
Learning the structure of Directed Acyclic Graphs (DAGs) presents a significant challenge due to the vast combinatorial search space of possible graphs, which scales exponentially…
Newton Method Revisited: Global Convergence Rates up to for Stepsize Schedules and Linesearch Procedures
Slavomír Hanzely, Farshed Abdukhakimov, Martin Takáč
This paper investigates the global convergence of stepsized Newton methods for convex functions with Hölder continuous Hessians or third derivatives. We propose several simple step…
Adaptive Optimization Algorithms for Machine Learning
Slavomír Hanzely
Machine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This di…
Sketch-and-Project Meets Newton Method: Global Convergence with Low-Rank Updates
Slavomír Hanzely
In this paper, we propose the first sketch-and-project Newton method with fast global convergence rate for self-concordant functions. Our method, SGN, can be v…
Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes
Konstantin Mishchenko, Slavomír Hanzely, Peter Richtárik
In this work, we consider the problem of minimizing the sum of Moreau envelopes of given functions, which has previously appeared in the context of meta-learning and personalized f…
A Damped Newton Method Achieves Global and Local Quadratic Convergence Rate
Slavomír Hanzely, Dmitry Kamzolov, Dmitry Pasechnyuk +3
In this paper, we present the first stepsize schedule for Newton method resulting in fast global and local convergence guarantees. In particular, a) we prove an $O\left( \frac 1 {k…
Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation
Rustem Islamov, Xun Qian, Slavomír Hanzely +2
Despite their high computation and communication costs, Newton-type methods remain an appealing option for distributed training due to their robustness against ill-conditioned conv…
Lower Bounds and Optimal Algorithms for Personalized Federated Learning
Filip Hanzely, Slavomír Hanzely, Samuel Horváth +1
In this work, we consider the optimization formulation of personalized federated learning recently introduced by Hanzely and Richtárik (2020) which was shown to give an alternative…