339 citations · 954 across the 23 of their papers we have counts for
15 papers · 1 filter
Invariant Layers for Graphs with Nodes of Different Types
Dmitry Rybin, Ruoyu Sun, Zhi-Quan Luo
Neural networks that satisfy invariance with respect to input permutations have been widely studied in machine learning literature. However, in many applications, only a subset of…
Adversarial Rademacher Complexity of Deep Neural Networks
Jiancong Xiao, Yanbo Fan, Ruoyu Sun +1
Deep neural networks (DNNs) are highly vulnerable to adversarial attacks. Ideally, a robust model should perform well on both perturbed training data and unseen perturbed test data…
Stability Analysis and Generalization Bounds of Adversarial Training
Jiancong Xiao, Yanbo Fan, Ruoyu Sun +2
In adversarial machine learning, deep neural networks can fit the adversarial examples on the training dataset but have poor generalization ability on the test set. This phenomenon…
When Expressivity Meets Trainability: Fewer than Neurons Can Work
Jiawei Zhang, Yushun Zhang, Mingyi Hong +2
Modern neural networks are often quite wide, causing large memory and computation costs. It is thus of great interest to train a narrower network. However, training narrow neural n…
Adam Can Converge Without Any Modification On Update Rules
Yushun Zhang, Congliang Chen, Naichen Shi +2
Ever since Reddi et al. 2018 pointed out the divergence issue of Adam, many new variants have been designed to obtain convergence. However, vanilla Adam remains exceptionally popul…
Towards Understanding the Impact of Model Size on Differential Private Classification
Yinchen Shen, Zhiguo Wang, Ruoyu Sun +1
Differential privacy (DP) is an essential technique for privacy-preserving. It was found that a large model trained for privacy preserving performs worse than a smaller model (e.g.…
Federated Semi-Supervised Learning with Class Distribution Mismatch
Zhiguo Wang, Xintong Wang, Ruoyu Sun +1
Many existing federated learning (FL) algorithms are designed for supervised learning tasks, assuming that the local data owned by the clients are well labeled. However, in many pr…
Achieving Small Test Error in Mildly Overparameterized Neural Networks
Shiyu Liang, Ruoyu Sun, R. Srikant
Recent theoretical works on over-parameterized neural nets have focused on two aspects: optimization and generalization. Many existing works that study optimization and generalizat…
On the Landscape of One-hidden-layer Sparse Networks and Beyond
Dachao Lin, Ruoyu Sun, Zhihua Zhang
Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused…
The Global Landscape of Neural Networks: An Overview
Ruoyu Sun, Dawei Li, Shiyu Liang +2
One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks sugg…
Optimization for deep learning: theory and algorithms
Ruoyu Sun
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss t…
Max-Sliced Wasserstein Distance and its use for GANs
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun +6
Generative adversarial nets (GANs) and variational auto-encoders have significantly improved our distribution modeling capabilities, showing promise for dataset augmentation, image…
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
Xiangyi Chen, Sijia Liu, Ruoyu Sun +1
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, whic…
Understanding the Loss Surface of Neural Networks for Binary Classification
Shiyu Liang, Ruoyu Sun, Yixuan Li +1
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun e…
Guaranteed Matrix Completion via Non-convex Factorization
Ruoyu Sun, Zhi-Quan Luo
Matrix factorization is a popular approach for large-scale matrix completion. The optimization formulation based on matrix factorization can be solved very efficiently by standard…