2 citations · 2 across the 13 of their papers we have counts for
13 papers
Mini-Batch Stochastic Krasnosel'ski\uı-Mann Algorithm for Nonexpansive Fixed Point Problems
Hideaki Iiduka
The Krasnosel'ski\uı-Mann algorithm is a well-known method for finding fixed points of a nonexpansive mapping with strong theoretical guarantees. However, there are practical large…
Mini-Batch Stochastic Halpern Algorithm for Nonexpansive Fixed Point Problems
Hideaki Iiduka
The Halpern algorithm is a powerful fixed point approximation method for finding the closest point in the fixed point set of a nonexpansive mapping to the initial point. However, i…
Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity
Hikaru Umeda, Hideaki Iiduka
The convergence behavior of mini-batch stochastic gradient descent (SGD) is highly sensitive to the batch size and learning rate settings. Recent theoretical studies have identifie…
Optimal Growth Schedules for Batch Size and Learning Rate in SGD that Reduce SFO Complexity
Hikaru Umeda, Hideaki Iiduka
The unprecedented growth of deep learning models has enabled remarkable advances but introduced substantial computational bottlenecks. A key factor contributing to training efficie…
Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis
Yuichi Kondo, Hideaki Iiduka
We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyap…
Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Size
Kento Imaizumi, Hideaki Iiduka
Momentum methods were originally introduced for their superiority to stochastic gradient descent (SGD) in deterministic settings with convex objective functions. However, despite t…
Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
Kanata Oowada, Hideaki Iiduka
We theoretically analyzed the convergence behavior of Riemannian stochastic gradient descent (RSGD) and found that using an increasing batch size leads to faster convergence than u…
Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum
Keisuke Kamo, Hideaki Iiduka
Stochastic gradient descent with momentum (SGDM), in which a momentum term is added to SGD, has been well studied in both theory and practice. The theoretical studies show that the…
Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
Hinata Harada, Hideaki Iiduka
The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural n…
Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
Hikaru Umeda, Hideaki Iiduka
The performance of mini-batch stochastic gradient descent (SGD) strongly depends on setting the batch size and learning rate to minimize the empirical loss in training the deep neu…
A general framework of Riemannian adaptive optimization methods with a convergence analysis
Hiroyuki Sakai, Hideaki Iiduka
This paper proposes a general framework of Riemannian adaptive optimization methods. The framework encapsulates several stochastic optimization algorithms on Riemannian manifolds a…
Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates
Kento Imaizumi, Hideaki Iiduka
The performance of stochastic gradient descent (SGD), which is the simplest first-order optimizer for training deep neural networks, depends on not only the learning rate but also…
Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifold
Hiroyuki Sakai, Hideaki Iiduka
Novel convergence analyses are presented of Riemannian stochastic gradient descent (RSGD) on a Hadamard manifold. RSGD is the most basic Riemannian stochastic optimization algorith…