339 citations · 954 across the 23 of their papers we have counts for
24 papers
Invariant Layers for Graphs with Nodes of Different Types
Dmitry Rybin, Ruoyu Sun, Zhi-Quan Luo
Neural networks that satisfy invariance with respect to input permutations have been widely studied in machine learning literature. However, in many applications, only a subset of…
Adversarial Rademacher Complexity of Deep Neural Networks
Jiancong Xiao, Yanbo Fan, Ruoyu Sun +1
Deep neural networks (DNNs) are highly vulnerable to adversarial attacks. Ideally, a robust model should perform well on both perturbed training data and unseen perturbed test data…
Stability Analysis and Generalization Bounds of Adversarial Training
Jiancong Xiao, Yanbo Fan, Ruoyu Sun +2
In adversarial machine learning, deep neural networks can fit the adversarial examples on the training dataset but have poor generalization ability on the test set. This phenomenon…
When Expressivity Meets Trainability: Fewer than Neurons Can Work
Jiawei Zhang, Yushun Zhang, Mingyi Hong +2
Modern neural networks are often quite wide, causing large memory and computation costs. It is thus of great interest to train a narrower network. However, training narrow neural n…
Adam Can Converge Without Any Modification On Update Rules
Yushun Zhang, Congliang Chen, Naichen Shi +2
Ever since Reddi et al. 2018 pointed out the divergence issue of Adam, many new variants have been designed to obtain convergence. However, vanilla Adam remains exceptionally popul…
Towards Understanding the Impact of Model Size on Differential Private Classification
Yinchen Shen, Zhiguo Wang, Ruoyu Sun +1
Differential privacy (DP) is an essential technique for privacy-preserving. It was found that a large model trained for privacy preserving performs worse than a smaller model (e.g.…
Federated Semi-Supervised Learning with Class Distribution Mismatch
Zhiguo Wang, Xintong Wang, Ruoyu Sun +1
Many existing federated learning (FL) algorithms are designed for supervised learning tasks, assuming that the local data owned by the clients are well labeled. However, in many pr…
Achieving Small Test Error in Mildly Overparameterized Neural Networks
Shiyu Liang, Ruoyu Sun, R. Srikant
Recent theoretical works on over-parameterized neural nets have focused on two aspects: optimization and generalization. Many existing works that study optimization and generalizat…
On the Landscape of One-hidden-layer Sparse Networks and Beyond
Dachao Lin, Ruoyu Sun, Zhihua Zhang
Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused…
The Global Landscape of Neural Networks: An Overview
Ruoyu Sun, Dawei Li, Shiyu Liang +2
One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks sugg…
Optimization for deep learning: theory and algorithms
Ruoyu Sun
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss t…
Max-Sliced Wasserstein Distance and its use for GANs
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun +6
Generative adversarial nets (GANs) and variational auto-encoders have significantly improved our distribution modeling capabilities, showing promise for dataset augmentation, image…
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
Xiangyi Chen, Sijia Liu, Ruoyu Sun +1
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, whic…
Adding One Neuron Can Eliminate All Bad Local Minima
Shiyu Liang, Ruoyu Sun, Jason D. Lee +1
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of n…
Understanding the Loss Surface of Neural Networks for Binary Classification
Shiyu Liang, Ruoyu Sun, Yixuan Li +1
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun e…
Training Language Models Using Target-Propagation
Sam Wiseman, Sumit Chopra, Marc'Aurelio Ranzato +4
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making…
Worst-case Complexity of Cyclic Coordinate Descent: Gap with Randomized Version
Ruoyu Sun, Yinyu Ye
This paper concerns the worst-case complexity of cyclic coordinate descent (C-CD) for minimizing a convex quadratic function, which is equivalent to Gauss-Seidel method and can be…
Improved Iteration Complexity Bounds of Cyclic Block Coordinate Descent for Convex Problems
Ruoyu Sun, Mingyi Hong
The iteration complexity of the block-coordinate descent (BCD) type algorithm has been under extensive investigation. It was recently shown that for convex problems the classical c…
On the Efficiency of Random Permutation for ADMM and Coordinate Descent
Ruoyu Sun, Zhi-Quan Luo, Yinyu Ye
Random permutation is observed to be powerful for optimization algorithms: for multi-block ADMM (alternating direction method of multipliers), while the classical cyclic version di…
Guaranteed Matrix Completion via Non-convex Factorization
Ruoyu Sun, Zhi-Quan Luo
Matrix factorization is a popular approach for large-scale matrix completion. The optimization formulation based on matrix factorization can be solved very efficiently by standard…
Joint Downlink Base Station Association and Power Control for Max-Min Fairness: Computation and Complexity
Ruoyu Sun, Mingyi Hong, Zhi-Quan Luo
In a heterogeneous network (HetNet) with a large number of low power base stations (BSs), proper user-BS association and power control is crucial to achieving desirable system perf…
Cross Layer Provision of Future Cellular Networks
H. Baligh, M. Hong, W. -C. Liao +4
To cope with the growing demand for wireless data and to extend service coverage, future 5G networks will increasingly rely on the use of low powered nodes to support massive conne…
Interference alignment using finite and dependent channel extensions: the single beam case
Ruoyu Sun, Zhi-Quan Luo
Vector space interference alignment (IA) is known to achieve high degrees of freedom (DoF) with infinite independent channel extensions, but its performance is largely unknown for…
Joint Base Station Clustering and Beamformer Design for Partial Coordinated Transmission in Heterogenous Networks
Mingyi Hong, Ruo-Yu Sun, Hadi Baligh +1
We consider the interference management problem in a multicell MIMO heterogenous network. Within each cell there are a large number of distributed micro/pico base stations (BSs) th…