979 citations · 1.5k across the 63 of their papers we have counts for
27 papers · 1 filter
Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis
Dachao Lin, Yuze Han, Haishan Ye +1
We study finite-sum distributed optimization problems involving a master node and local nodes under the popular -similarity and -strong convexity conditions. We propose…
Federated Reinforcement Learning with Environment Heterogeneity
Hao Jin, Yang Peng, Wenhao Yang +2
We study a Federated Reinforcement Learning (FedRL) problem in which agents collaboratively learn a single policy without sharing the trajectories they collected during agent-e…
Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer
Kun Chen, Dachao Lin, Zhihua Zhang
In this paper, we follow Eftekhari's work to give a non-local convergence analysis of deep linear networks. Specifically, we consider optimizing deep linear networks which have a l…
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen +3
Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Transformers is memory-intensive, which renders the searc…
Directional Convergence Analysis under Spherically Symmetric Distribution
Dachao Lin, Zhihua Zhang
We consider the fundamental problem of learning linear predictors (i.e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent. Under th…
Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
Guangzeng Xie, Hao Jin, Dachao Lin +1
We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective…
DIPPA: An improved Method for Bilinear Saddle Point Problems
Guangzeng Xie, Yuze Han, Zhihua Zhang
This paper studies bilinear saddle point problems , where the functions are smooth and stron…
Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
Wenhao Yang, Xiang Li, Guangzeng Xie +1
Regularized MDPs serve as a smooth version of original MDPs. However, biased optimal policy always exists for regularized MDPs. Instead of making the coefficientλof regularized ter…
On the Landscape of One-hidden-layer Sparse Networks and Beyond
Dachao Lin, Ruoyu Sun, Zhihua Zhang
Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused…
Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
Dachao Lin, Peiqin Sun, Guangzeng Xie +2
Quantized Neural Networks (QNNs) use low bit-width fixed-point numbers for representing weight parameters and activations, and are often used in real-world applications due to thei…
Fast Generalized Matrix Regression with Applications in Machine Learning
Haishan Ye, Shusen Wang, Zhihua Zhang +1
Fast matrix algorithms have become the fundamental tools of machine learning in big data era. The generalized matrix regression problem is widely used in the matrix approximation s…
A Stochastic Proximal Point Algorithm for Saddle-Point Problems
Luo Luo, Cheng Chen, Yujun Li +2
We consider saddle point problems which objective functions are the average of strongly convex-concave individual components. Recently, researchers exploit variance reduction m…
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
Tianle Cai, Ruiqi Gao, Jikai Hou +5
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better con…
Lipschitz Generative Adversarial Nets
Zhiming Zhou, Jiadong Liang, Yuxuan Song +5
In this paper, we study the convergence of generative adversarial networks (GANs) from the perspective of the informativeness of the gradient of the optimal discriminative function…
Nesterov's Acceleration For Approximate Newton
Haishan Ye, Zhihua Zhang
Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. Ho…
Nestrov's Acceleration For Second Order Method
Haishan Ye, Zhihua Zhang
Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. Ho…
Robust Frequent Directions with Application in Online Learning
Luo Luo, Cheng Chen, Zhihua Zhang +2
The frequent directions (FD) technique is a deterministic approach for online sketching that has many applications in machine learning. The conventional FD is a heuristic procedure…
Communication Lower Bounds for Distributed Convex Optimization: Partition Data on Features
Zihao Chen, Luo Luo, Zhihua Zhang
Recently, there has been an increasing interest in designing distributed convex optimization algorithms under the setting where the data matrix is partitioned on features. Algorith…
A Proximal Stochastic Quasi-Newton Algorithm
Luo Luo, Zihao Chen, Zhihua Zhang +1
In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the…
A New Relaxation Approach to Normalized Hypergraph Cut
Cong Xie, Wu-Jun Li, Zhihua Zhang
Normalized graph cut (NGC) has become a popular research topic due to its wide applications in a large variety of areas like machine learning and very large scale integration (VLSI…
Regret vs. Communication: Distributed Stochastic Multi-Armed Bandits and Beyond
Shuang Liu, Cheng Chen, Zhihua Zhang
In this paper, we consider the distributed stochastic multi-armed bandit problem, where a global arm set can be accessed by multiple players independently. The players are allowed…
Towards More Efficient SPSD Matrix Approximation and CUR Matrix Decomposition
Shusen Wang, Zhihua Zhang, Tong Zhang
Symmetric positive semi-definite (SPSD) matrix approximation methods have been extensively used to speed up large-scale eigenvalue computation and kernel learning methods. The stan…
Adjusting Leverage Scores by Row Weighting: A Practical Approach to Coherent Matrix Completion
Shusen Wang, Tong Zhang, Zhihua Zhang
Low-rank matrix completion is an important problem with extensive real-world applications. When observations are uniformly sampled from the underlying matrix entries, existing meth…
SPSD Matrix Approximation vis Column Selection: Theories, Algorithms, and Extensions
Shusen Wang, Luo Luo, Zhihua Zhang
Symmetric positive semidefinite (SPSD) matrix approximation is an important problem with applications in kernel methods. However, existing SPSD matrix approximation methods such as…