5 citations · 10 across the 21 of their papers we have counts for
22 papers
Optimal Parameter-Free Second-order Acceleration with Inexact Hessians under Generalized Smoothness
Artem Agafonov, Aslan Ashabokov, Alexander D'yakonov +3
We study second-order convex optimization under generalized Hessian smoothness, where the local Hessian Lipschitz constant may grow linearly with the gradient norm. For inexact Hes…
Certified Residual Quasi-Newton Methods for Distributed Variational Inequalities
Ewsey R. Obzherin, Roman M. Mozhaev, Alexander V. Gasnikov +3
Second-order methods for smooth monotone variational inequalities reach the optimal rate , but a distributed exact Jacobian costs times more communication than an…
Application of Optimal Inexact Second-Order Acceleration to Distributed Stochastic Optimization under Statistical Similarity
Yury A. Sokolov, Maxim K. Mashtaler, Alexander V. Gasnikov +3
We consider distributed stochastic convex optimization with a fixed budget of independent samples split among workers. Sample average approximation reduces the problem to a…
Neither Adversarial Training Nor Purification: Emergent Adversarial Robustness from Oscillatory Predictive Learning
Mohammed-Yassine Habibi, Klea Ziu, Martin Takáč +1
Adversarial robustness in computer vision is still largely achieved through adversarial training or test-time adversarial purification, both of which introduce significant computat…
Chebyshev-Exact Acceleration under Hessian Variation, I: Sine-Jacobi Method
Dmitry Pasechnyuk-Vilensky, Martin Takáč
We study finite-horizon one-gradient realizations with the Chebyshev minimax terminal residual on . Under time-dependent Hessian perturbations, the terminal first variation…
Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference
Alexander Yukhimchuk, Andrey Shulga, Mladen Kolar +1
Running large language models locally is often impractical, pushing inference on sensitive text to third-party providers. Split inference partially mitigates this by keeping tokens…
Decentralized Inexact Cubic Newton Method with Consensus Procedure
Artem Agafonov, Anton Novitskii, Alexander Rogozin +5
Distributed optimization is widely used in large-scale and privacy-preserving machine learning, where each agent stores a local objective and communicates only with its neighbors i…
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
Alexander Yukhimchuk, Mladen Kolar, Martin Takáč +1
Gradient clipping is a standard safeguard for training neural networks under noisy, heavy-tailed stochastic gradients; yet, most clipping rules treat all parameters as vectors and…
Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
Sayantan Choudhury, Xiaoran Cheng, Martin Takáč +2
Most first-order optimizers treat matrix-valued parameters as vectors, ignoring the intrinsic geometry of hidden-layer weights in neural networks. Muon addresses this mismatch by u…
Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
Artem Riabinin, Andrey Veprikov, Arman Bolatov +2
We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decre…
CORE: Collaborative Reasoning via Cross Teaching
Kshitij Mishra, Mirat Aubakirov, Martin Takac +2
Large language models exhibit complementary reasoning errors: on the same instance, one model may succeed with a particular decomposition while another fails. We propose Collaborat…
Learning Latent Graph Geometry via Fixed-Point Schrödinger-Type Activation: A Theoretical Study
Dmitry Pasechnyuk-Vilensky, Martin Takáč
We study neural architectures in which each hidden layer is defined by the stationary state of a dissipative Schrödinger-type dynamics on a learned latent graph. On stable branches…
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
Omar Choukrani, Idriss Malek, Daniil Orel +5
Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful f…
Bant: Byzantine Antidote via Trial Function and Trust Scores
Gleb Molodtsov, Daniil Medyakov, Sergey Skorik +6
Recent advancements in machine learning have improved performance while also increasing computational demands. While federated and distributed setups address these issues, their st…
Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens
Nikolai Rekut, Alexey Orlov, Klea Ziu +3
Representing molecular structures effectively in chemistry remains a challenging task. Language models and graph-based models are extensively utilized within this domain, consisten…
DAG: Projected Stochastic Approximation Iteration for DAG Structure Learning
Klea Ziu, Slavomír Hanzely, Loka Li +3
Learning the structure of Directed Acyclic Graphs (DAGs) presents a significant challenge due to the vast combinatorial search space of possible graphs, which scales exponentially…
OPTAMI: Global Superlinear Convergence of High-order Methods
Dmitry Kamzolov, Dmitry Pasechnyuk, Artem Agafonov +2
Second-order methods for convex optimization outperform first-order methods in terms of theoretical iteration convergence, achieving rates up to for highly-smooth funct…
Local Methods with Adaptivity via Scaling
Savelii Chezhegov, Sergey Skorik, Nikolas Khachaturov +5
The rapid development of machine learning and deep learning has introduced increasingly complex optimization challenges that must be addressed. Indeed, training modern, advanced mo…
SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms
Farshed Abdukhakimov, Chulu Xiang, Dmitry Kamzolov +2
Adaptive optimization methods are widely recognized as among the most popular approaches for training Deep Neural Networks (DNNs). Techniques such as Adam, AdaGrad, and AdaHessian…
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
Aleksandr Beznosikov, Martin Takáč, Alexander Gasnikov
Variational inequalities are a broad and flexible class of problems that includes minimization, saddle point, and fixed point problems as special cases. Therefore, variational ineq…
FLECS: A Federated Learning Second-Order Framework via Compression and Sketching
Artem Agafonov, Dmitry Kamzolov, Rachael Tappenden +2
Inspired by the recent work FedNL (Safaryan et al, FedNL: Making Newton-Type Methods Applicable to Federated Learning), we propose a new communication efficient second-order framew…
Inexact Tensor Methods and Their Application to Stochastic Convex Optimization
Artem Agafonov, Dmitry Kamzolov, Pavel Dvurechensky +2
We propose general non-accelerated and accelerated tensor methods under inexact information on the derivatives of the objective, analyze their convergence rate. Further, we provide…