224 citations · 3.2k across the 170 of their papers we have counts for
95 papers · 1 filter
Memory Efficient Optimizers with 4-bit States
Bingrui Li, Jianfei Chen, Jun Zhu
Optimizer states are a major source of memory consumption for training neural networks, limiting the maximum trainable model within given memory budget. Compressing the optimizer s…
Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence
Liyuan Wang, Xingxing Zhang, Qian Li +4
Continual learning aims to empower artificial intelligence (AI) with strong adaptability to the real world. For this purpose, a desirable solution should properly balance memory st…
Towards Accelerated Model Training via Bayesian Data Selection
Zhijie Deng, Peng Cui, Jun Zhu
Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard sam…
MultiAdam: Parameter-wise Scale-invariant Optimizer for Multiscale Training of Physics-informed Neural Networks
Jiachen Yao, Chang Su, Zhongkai Hao +3
Physics-informed Neural Networks (PINNs) have recently achieved remarkable progress in solving Partial Differential Equations (PDEs) in various fields by minimizing a weighted sum…
PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs
Zhongkai Hao, Jiachen Yao, Chang Su +8
While significant progress has been made on Physics-Informed Neural Networks (PINNs), a comprehensive comparison of these methods across a wide range of Partial Differential Equati…
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Liyuan Wang, Xingxing Zhang, Hang Su +1
To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime. This ability, known as co…
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One
Liyuan Wang, Xingxing Zhang, Qian Li +2
Continual learning requires incremental compatibility with a sequence of tasks. However, the design of model architecture remains an open question: In general, learning all tasks w…
Thompson Sampling for (Combinatorial) Pure Exploration
Siwei Wang, Jun Zhu
Existing methods of combinatorial pure exploration mainly focus on the UCB approach. To make the algorithm efficient, they usually use the sum of upper confidence bounds within arm…
NeuralEF: Deconstructing Kernels by Deep Neural Networks
Zhijie Deng, Jiaxin Shi, Jun Zhu
Learning the principal eigenfunctions of an integral operator defined by a kernel and a data distribution is at the core of many machine learning problems. Traditional nonparametri…
Deep Ensemble as a Gaussian Process Approximate Posterior
Zhijie Deng, Feng Zhou, Jianfei Chen +2
Deep Ensemble (DE) is an effective alternative to Bayesian neural networks for uncertainty quantification in deep learning. The uncertainty of DE is usually conveyed by the functio…
Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Prior
Yinpeng Dong, Shuyu Cheng, Tianyu Pang +2
Adversarial attacks have been extensively studied in recent years since they can identify the vulnerability of deep learning models before deployed. In this paper, we consider the…
AFEC: Active Forgetting of Negative Transfer in Continual Learning
Liyuan Wang, Mingtian Zhang, Zhongfan Jia +5
Continual learning aims to learn a sequence of tasks from dynamic data distributions. Without accessing to the old training samples, knowledge transfer from the old tasks to each n…
Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
Xiao Yang, Yinpeng Dong, Wenzhao Xiang +3
The vulnerability of deep neural networks to adversarial examples has motivated an increasing number of defense strategies for promoting model robustness. However, the progress is…
Tianshou: a Highly Modularized Deep Reinforcement Learning Library
Jiayi Weng, Huayu Chen, Dong Yan +6
In this paper, we present Tianshou, a highly modularized Python library for deep reinforcement learning (DRL) that uses PyTorch as its backend. Tianshou intends to be research-frie…
Boosting Transferability of Targeted Adversarial Examples via Hierarchical Generative Networks
Xiao Yang, Yinpeng Dong, Tianyu Pang +2
Transfer-based adversarial attacks can evaluate model robustness in the black-box setting. Several methods have demonstrated impressive untargeted transferability, however, it is s…
Regularized OFU: an Efficient UCB Estimator forNon-linear Contextual Bandit
Yichi Zhou, Shihong Song, Huishuai Zhang +3
Balancing exploration and exploitation (EE) is a fundamental problem in contex-tual bandit. One powerful principle for EE trade-off isOptimism in Face of Uncer-tainty(OFU), in whic…
Understanding Softmax Confidence and Uncertainty
Tim Pearce, Alexandra Brintrup, Jun Zhu
It is often remarked that neural networks fail to increase their uncertainty when predicting on data far from the training distribution. Yet naively using softmax confidence as a p…
Accumulative Poisoning Attacks on Real-time Data
Tianyu Pang, Xiao Yang, Yinpeng Dong +2
Collecting training data from untrusted sources exposes machine learning services to poisoning adversaries, who maliciously manipulate training data to degrade the model accuracy.…
Stability and Generalization of Bilevel Programming in Hyperparameter Optimization
Fan Bao, Guoqiang Wu, Chongxuan Li +2
The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainl…
Exploring Memorization in Adversarial Training
Yinpeng Dong, Ke Xu, Xiao Yang +4
Deep learning models have a propensity for fitting the entire training set even with random labels, which requires memorization of every training sample. In this paper, we explore…
Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations
Yingtao Luo, Qiang Liu, Yuntian Chen +3
Partial differential equations (PDEs) that fit scientific data can represent physical laws with explainable mechanisms for various mathematically-oriented subjects, such as physics…
Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization
Guoqiang Wu, Chongxuan Li, Kun Xu +1
(Partial) ranking loss is a commonly used evaluation measure for multi-label classification, which is usually optimized with convex surrogates for computational efficiency. Prior t…
Automated Decision-based Adversarial Attacks
Qi-An Fu, Yinpeng Dong, Hang Su +1
Deep learning models are vulnerable to adversarial examples, which can fool a target classifier by imposing imperceptible perturbations onto natural examples. In this work, we cons…
MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
Tsung Wei Tsai, Chongxuan Li, Jun Zhu
We present Mixture of Contrastive Experts (MiCE), a unified probabilistic clustering framework that simultaneously exploits the discriminative representations learned by contrastiv…