979 citations · 1.5k across the 63 of their papers we have counts for
63 papers
Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis
Dachao Lin, Yuze Han, Haishan Ye +1
We study finite-sum distributed optimization problems involving a master node and local nodes under the popular -similarity and -strong convexity conditions. We propose…
Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Data
Xiang Li, Jiadong Liang, Zhihua Zhang
Many stochastic approximation (SA) algorithms evolve along a single trajectory, making uncertainty quantification challenging under nonlinear dynamics and Markov dependence. We dev…
Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach
Miao Lu, Wenhao Yang, Liangyu Zhang +1
In an Markov decision process (MDP), unobservable confounders may exist and have impacts on the data generating process, so that the classic off-policy evaluation (OPE) estimators…
On the Convergence of Policy in Unregularized Policy Mirror Descent
Dachao Lin, Zhihua Zhang
In this short note, we give the convergence analysis of the policy in the recent famous policy mirror descent (PMD). We mainly consider the unregularized setting following [11] wit…
Federated Reinforcement Learning with Environment Heterogeneity
Hao Jin, Yang Peng, Wenhao Yang +2
We study a Federated Reinforcement Learning (FedRL) problem in which agents collaboratively learn a single policy without sharing the trajectories they collected during agent-e…
Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer
Kun Chen, Dachao Lin, Zhihua Zhang
In this paper, we follow Eftekhari's work to give a non-local convergence analysis of deep linear networks. Specifically, we consider optimizing deep linear networks which have a l…
Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations
Haishan Ye, Dachao Lin, Zhihua Zhang
In this paper, we propose the greedy and random Broyden's method for solving nonlinear equations. Specifically, the greedy method greedily selects the direction to maximize a certa…
Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations
Dachao Lin, Haishan Ye, Zhihua Zhang
In this paper, we study the explicit superlinear convergence rates of quasi-Newton methods. We particularly focus on the classical Broyden's method for solving nonlinear equations.…
Statistical Estimation and Inference via Local SGD in Federated Learning
Xiang Li, Jiadong Liang, Xiangyu Chang +1
Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decen…
Explicit Superlinear Convergence Rates of The SR1 Algorithm
Haishan Ye, Dachao Lin, Zhihua Zhang +1
We study the convergence rate of the famous Symmetric Rank-1 (SR1) algorithm which has wide applications in different scenarios. Although it has been extensively investigated, SR1…
Near Optimal Stochastic Algorithms for Finite-Sum Unbalanced Convex-Concave Minimax Optimization
Luo Luo, Guangzeng Xie, Tong Zhang +1
This paper considers stochastic first-order algorithms for convex-concave minimax problems of the form , where can be presented by the…
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen +3
Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Transformers is memory-intensive, which renders the searc…
Directional Convergence Analysis under Spherically Symmetric Distribution
Dachao Lin, Zhihua Zhang
We consider the fundamental problem of learning linear predictors (i.e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent. Under th…
Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
Guangzeng Xie, Hao Jin, Dachao Lin +1
We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective…
Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods
Dachao Lin, Haishan Ye, Zhihua Zhang
Optimization is important in machine learning problems, and quasi-Newton methods have a reputation as the most efficient numerical schemes for smooth unconstrained optimization. In…
DIPPA: An improved Method for Bilinear Saddle Point Problems
Guangzeng Xie, Yuze Han, Zhihua Zhang
This paper studies bilinear saddle point problems , where the functions are smooth and stron…
Lower Complexity Bounds of Finite-Sum Optimization Problems: The Results and Construction
Yuze Han, Guangzeng Xie, Zhihua Zhang
In this paper, we study the lower complexity bounds for finite-sum optimization problems, where the objective is the average of individual component functions. We consider Prox…
Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applications
Xiang Li, Zhihua Zhang
In this work, we study a novel class of projection-based algorithms for linearly constrained problems (LCPs) which have a lot of applications in statistics, optimization, and machi…
Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
Wenhao Yang, Xiang Li, Guangzeng Xie +1
Regularized MDPs serve as a smooth version of original MDPs. However, biased optimal policy always exists for regularized MDPs. Instead of making the coefficientλof regularized ter…
On the Landscape of One-hidden-layer Sparse Networks and Beyond
Dachao Lin, Ruoyu Sun, Zhihua Zhang
Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused…
Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
Dachao Lin, Peiqin Sun, Guangzeng Xie +2
Quantized Neural Networks (QNNs) use low bit-width fixed-point numbers for representing weight parameters and activations, and are often used in real-world applications due to thei…
Intervention Generative Adversarial Networks
Jiadong Liang, Liangyu Zhang, Cheng Zhang +1
In this paper we propose a novel approach for stabilizing the training process of Generative Adversarial Networks as well as alleviating the mode collapse problem. The main idea is…
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
Yimin Huang, Yujun Li, Hanrong Ye +2
The evaluation of hyperparameters, neural architectures, or data augmentation policies becomes a critical model selection problem in advanced deep learning with a large hyperparame…
Communication-Efficient Distributed SVD via Local Power Iterations
Xiang Li, Shusen Wang, Kun Chen +1
We study distributed computing of the truncated singular value decomposition problem. We develop an algorithm that we call \texttt{LocalPower} for improving communication efficienc…