5 citations · 16 across the 22 of their papers we have counts for
24 papers
Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization
Naoki Sato, Hideaki Iiduka
The graduated optimization approach is a method for finding global optimal solutions for nonconvex functions by using a function smoothing operation with stochastic noise. This pap…
Relationship between Batch Size and Number of Steps Needed for Nonconvex Optimization of Stochastic Gradient Descent using Armijo Line Search
Yuki Tsukada, Hideaki Iiduka
While stochastic gradient descent (SGD) can use various learning rates, such as constant or diminishing rates, the previous numerical results showed that SGD performs better than o…
Modified memoryless spectral-scaling Broyden family on Riemannian manifolds
Hiroyuki Sakai, Hideaki Iiduka
This paper presents modified memoryless quasi-Newton methods based on the spectral-scaling Broyden family on Riemannian manifolds. The method involves adding one parameter to the s…
Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One
Hideaki Iiduka
Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can find the model parameters of…
Global Convergence of Hager-Zhang type Riemannian Conjugate Gradient Method
Hiroyuki Sakai, Hiroyuki Sato, Hideaki Iiduka
This paper presents the Hager-Zhang (HZ)-type Riemannian conjugate gradient method that uses the exponential retraction. We also present global convergence analyses of our proposed…
Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness
Hideaki Iiduka
Convergence and convergence rate analyses of adaptive methods, such as Adaptive Moment Estimation (Adam) and its variants, have been widely studied for nonconvex optimization. The…
Conjugate Gradient Method for Generative Adversarial Networks
Hiroki Naganuma, Hideaki Iiduka
One of the training strategies of generative models is to minimize the Jensen--Shannon divergence between the model distribution and the data distribution. Since data distribution…
Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule
Naoki Sato, Hideaki Iiduka
Previous results have shown that a two time-scale update rule (TTUR) using different learning rates, such as different constant rates or different decaying rates, is useful for tra…
Minimization of Stochastic First-order Oracle Complexity of Adaptive Methods for Nonconvex Optimization
Hideaki Iiduka
Numerical evaluations have definitively shown that, for deep learning optimizers such as stochastic gradient descent, momentum, and adaptive methods, the number of steps needed to…
The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size
Hideaki Iiduka
Recently, convergence as well as convergence rate analyses of deep learning optimizers for nonconvex optimization have been widely studied. Meanwhile, numerical evaluations for the…
Riemannian Stochastic Fixed Point Optimization Algorithm
Hideaki Iiduka, Hiroyuki Sakai
This paper considers a stochastic optimization problem over the fixed point sets of quasinonexpansive mappings on Riemannian manifolds. The problem enables us to consider Riemannia…
Sufficient Descent Riemannian Conjugate Gradient Method
Hiroyuki Sakai, Hideaki Iiduka
This paper considers sufficient descent Riemannian conjugate gradient methods with line search algorithms. We propose two kinds of sufficient descent nonlinear conjugate gradient m…
Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing
Hiroyuki Sakai, Hideaki Iiduka
This paper proposes a Riemannian adaptive optimization algorithm to optimize the parameters of deep neural networks. The algorithm is an extension of both AMSGrad in Euclidean spac…
Conjugate-gradient-based Adam for stochastic optimization and its application to deep learning
Yu Kobayashi, Hideaki Iiduka
This paper proposes a conjugate-gradient-based Adam algorithm blending Adam with nonlinear conjugate gradient methods and shows its convergence analysis. Numerical experiments on t…
Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks
Hideaki Iiduka
This paper deals with nonconvex stochastic optimization problems in deep learning and provides appropriate learning rates with which adaptive learning rate optimization algorithms,…
Hybrid Riemannian Conjugate Gradient Methods with Global Convergence Properties
Hiroyuki Sakai, Hideaki Iiduka
This paper presents new Riemannian conjugate gradient methods and global convergence analyses under the strong Wolfe conditions. The main idea of the new methods is to combine the…
Fixed Point Quasiconvex Subgradient Method
Kazuhiro Hishinuma, Hideaki Iiduka
Constrained quasiconvex optimization problems appear in many fields, such as economics, engineering, and management science. In particular, fractional programming, which models rat…
Fixed Point Algorithm for Solving Nonmonotone Variational Inequalities in Nonnegative Matrix Factorization
Hideaki Iiduka, Shizuka Nishino
Nonnegative matrix factorization (NMF), which is the approximation of a data matrix as the product of two nonnegative matrices, is a key issue in machine learning and data analysis…
Incremental and Parallel Machine Learning Algorithms with Automated Learning Rate Adjustments
Kazuhiro Hishinuma, Hideaki Iiduka
The existing machine learning algorithms for minimizing the convex function over a closed convex set suffer from slow convergence because their learning rates must be determined be…
Two Stochastic Optimization Algorithms for Convex Optimization with Fixed Point Constraints
Hideaki Iiduka
Two optimization algorithms are proposed for solving a stochastic programming problem for which the objective function is given in the form of the expectation of convex functions a…
Proximal Point Algorithms for Nonsmooth Convex Optimization with Fixed Point Constraints
Hideaki Iiduka
The problem of minimizing the sum of nonsmooth, convex objective functions defined on a real Hilbert space over the intersection of fixed point sets of nonexpansive mappings, onto…
Almost Sure Convergence of Random Projected Proximal and Subgradient Algorithms for Distributed Nonsmooth Convex Optimization
Hideaki Iiduka
Two distributed algorithms are described that enable all users connected over a network to cooperatively solve the problem of minimizing the sum of all users' objective functions o…
Convergence Analysis of Iterative Methods for Nonsmooth Convex Optimization over Fixed Point Sets of Quasi-Nonexpansive Mappings
Hideaki Iiduka
This paper considers a networked system with a finite number of users and supposes that each user tries to minimize its own private objective function over its own private constrai…
Line Search Fixed Point Algorithms Based on Nonlinear Conjugate Gradient Directions: Application to Constrained Smooth Convex Optimization
Hideaki Iiduka
This paper considers the fixed point problem for a nonexpansive mapping on a real Hilbert space and proposes novel line search fixed point algorithms to accelerate the search. The…