198 citations · 775 across the 43 of their papers we have counts for
25 papers · 1 filter
Mirror Natural Evolution Strategies
Haishan Ye
The zeroth-order optimization has been widely used in machine learning applications. However, the theoretical study of the zeroth-order optimization focus on the algorithms which a…
IDEA: Interpretable Dynamic Ensemble Architecture for Time Series Prediction
Mengyue Zha, Kani Chen, Tong Zhang
We enhance the accuracy and generalization of univariate time series point prediction by an explainable ensemble on the fly. We propose an Interpretable Dynamic Ensemble Architectu…
Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Xiao Zhou, Weizhong Zhang, Zonghao Chen +2
Sparse training is a natural idea to accelerate the training speed of deep neural networks and save the memory usage, especially since large modern neural networks are significantl…
When is the Convergence Time of Langevin Algorithms Dimension Independent? A Composite Optimization Viewpoint
Yoav Freund, Yi-An Ma, Tong Zhang
There has been a surge of works bridging MCMC sampling and optimization, with a specific focus on translating non-asymptotic convergence guarantees for optimization problems into t…
Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learning
Tong Zhang
Thompson Sampling has been widely used for contextual bandit problems due to the flexibility of its modeling power. However, a general theory for this class of methods in the frequ…
Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums
Rui Pan, Haishan Ye, Tong Zhang
Learning rate schedulers have been widely adopted in training deep neural networks. Despite their practical importance, there is a discrepancy between its practice and its theoreti…
Feature Correlation Aggregation: on the Path to Better Graph Neural Networks
Jieming Zhou, Tong Zhang, Pengfei Fang +2
Prior to the introduction of Graph Neural Networks (GNNs), modeling and analyzing irregular data, particularly graphs, was thought to be the Achilles' heel of deep learning. The co…
Effective Sparsification of Neural Networks with Global Sparsity Constraint
Xiao Zhou, Weizhong Zhang, Hang Xu +1
Weight pruning is an effective technique to reduce the model size and inference time for deep neural networks in real-world deployments. However, since magnitudes and relative impo…
DeEPCA: Decentralized Exact PCA with Linear Convergence Rate
Haishan Ye, Tong Zhang
Due to the rapid growth of smart agents such as weakly connected computational nodes and sensors, developing decentralized algorithms that can perform computations on local agents…
Mathematical Models of Overparameterized Neural Networks
Cong Fang, Hanze Dong, Tong Zhang
Deep learning has received considerable empirical successes in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been…
Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information
Qi Chang, Zhennan Yan, Lohendran Baskaran +5
As deep learning technologies advance, increasingly more data is necessary to generate general and robust models for various tasks. In the medical domain, however, large-scale and…
DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks
Yuhui Ding, Quanming Yao, Huan Zhao +1
In this paper, we propose a novel framework to automatically utilize task-dependent semantic information which is encoded in heterogeneous information networks (HINs). Specifically…
Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization
Rie Johnson, Tong Zhang
This paper presents a framework of successive functional gradient optimization for training nonconvex models such as neural networks, where training is driven by mirror descent in…
Multi-consensus Decentralized Accelerated Gradient Descent
Haishan Ye, Luo Luo, Ziang Zhou +1
This paper considers the decentralized convex optimization problem, which has a wide range of applications in large-scale machine learning, sensor networks, and control theory. We…
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
Chaoyang He, Haishan Ye, Li Shen +1
Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of…
Bidirectional Generative Modeling Using Adversarial Gradient Estimation
Xinwei Shen, Tong Zhang, Kani Chen
This paper considers the general -divergence formulation of bidirectional generative modeling, which includes VAE and BiGAN as special cases. We present a new optimization metho…
A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
Zixiang Chen, Yuan Cao, Quanquan Gu +1
A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tan…
Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems
Luo Luo, Haishan Ye, Zhichao Huang +1
We consider nonconvex-concave minimax optimization problems of the form , where is strongly-concave in but p…
Fast Generalized Matrix Regression with Applications in Machine Learning
Haishan Ye, Shusen Wang, Zhihua Zhang +1
Fast matrix algorithms have become the fundamental tools of machine learning in big data era. The generalized matrix regression problem is widely used in the matrix approximation s…
Stable Learning via Sample Reweighting
Zheyan Shen, Peng Cui, Tong Zhang +1
We consider the problem of learning linear prediction models with model misspecification bias. In such case, the collinearity among input variables may inflate the error of paramet…
A Fast Sampling Gradient Tree Boosting Framework
Daniel Chao Zhou, Zhongming Jin, Tong Zhang
As an adaptive, interpretable, robust, and accurate meta-algorithm for arbitrary differentiable loss functions, gradient tree boosting is one of the most popular machine learning t…
Convex Formulation of Overparameterized Deep Neural Networks
Cong Fang, Yihong Gu, Weizhong Zhang +1
Analysis of over-parameterized neural networks has drawn significant attention in recentyears. It was shown that such systems behave like convex systems under various restrictedset…
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
Cong Fang, Hanze Dong, Tong Zhang
Recently, over-parameterized neural networks have been extensively analyzed in the literature. However, the previous studies cannot satisfactorily explain why fully trained neural…
A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization
Shun Zheng, Jialei Wang, Fen Xia +2
In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the "data parallelism" appro…