Wide Residual Networks

Abstract

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at https://github.com/szagoruyko/wide-residual-networks

References in corpus (6)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Deeply-Supervised Nets
• On the Number of Linear Regions of Deep Neural Networks
• Highway Networks
• Fractional Max-Pooling
• Deep Networks with Stochastic Depth

Cited by in corpus (921)

• Denoising Diffusion Probabilistic Models
• Neural Architecture Search with Reinforcement Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• Densely Connected Convolutional Networks
• On Calibration of Modern Neural Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Scaling Laws for Neural Language Models
• Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
• Score-Based Generative Modeling through Stochastic Differential Equations
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Interpolation Consistency Training for Semi-Supervised Learning
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• A Convergence Theory for Deep Learning via Over-Parameterization
• Embedding Watermarks into Deep Neural Networks
• Deep Learning in Video Multi-Object Tracking: A Survey
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Don't Decay the Learning Rate, Increase the Batch Size
• Group Equivariant Convolutional Networks
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Feature Pyramid Networks for Object Detection
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Energy-based Out-of-distribution Detection
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• Born Again Neural Networks
• Gabor Convolutional Networks
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Deep Anomaly Detection Using Geometric Transformations
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Aggregated Residual Transformations for Deep Neural Networks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• DeepViT: Towards Deeper Vision Transformer
• CBAM: Convolutional Block Attention Module
• Bayesian Compression for Deep Learning
• Recent Advances in Convolutional Neural Networks
• Systematic evaluation of CNN advances on the ImageNet
• Residual Attention Network for Image Classification
• Generating Adversarial Examples with Adversarial Networks
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• GridMask Data Augmentation
• Averaging Weights Leads to Wider Optima and Better Generalization
• Hierarchical Representations for Efficient Architecture Search
• On the Reconstruction of Face Images from Deep Face Templates
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• Revisiting ResNets: Improved Training and Scaling Strategies
• Adversarial Weight Perturbation Helps Robust Generalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Neural Optimizer Search with Reinforcement Learning
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Sparse Networks from Scratch: Faster Training without Losing Performance
• WRPN: Wide Reduced-Precision Networks
• RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
• Selective Classification for Deep Neural Networks
• Automatic Skin Lesion Analysis using Large-scale Dermoscopy Images and Deep Residual Networks
• Are Labels Required for Improving Adversarial Robustness?
• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Improving Variational Inference with Inverse Autoregressive Flow
• Bringing AI To Edge: From Deep Learning's Perspective
• Deep Complex Networks
• SpinalNet: Deep Neural Network with Gradual Input
• Invertible Residual Networks
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• A Light CNN for Deep Face Representation with Noisy Labels
• Learning Sparse Neural Networks through $L_0$ Regularization
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Context Encoding for Semantic Segmentation
• Scaled-YOLOv4: Scaling Cross Stage Partial Network
• Galaxy Morphology Classification with Deep Convolutional Neural Networks
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Scaling provable adversarial defenses
• Knowledge Distillation in Deep Learning and its Applications
• VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation
• Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
• RobustBench: a standardized adversarial robustness benchmark
• Fixup Initialization: Residual Learning Without Normalization
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Streaming convolutional neural networks for end-to-end learning with multi-megapixel images
• Wavelet Convolutional Neural Networks
• Varifocal-Net: A Chromosome Classification Approach using Deep Convolutional Networks
• Student-Teacher Feature Pyramid Matching for Anomaly Detection
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation
• FMix: Enhancing Mixed Sample Data Augmentation
• From Variational to Deterministic Autoencoders
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Denoising Diffusion Implicit Models
• Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
• Deep Convolutional Networks as shallow Gaussian Processes
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Squared Earth Mover's Distance-based Loss for Training Deep Neural Networks
• Training Quantized Nets: A Deeper Understanding
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Selection via Proxy: Efficient Data Selection for Deep Learning
• Dissecting Neural ODEs
• Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
• Self-Adaptive Training: beyond Empirical Risk Minimization
• On the Origin of Deep Learning
• Efficient Facial Representations for Age, Gender and Identity Recognition in Organizing Photo Albums using Multi-output CNN
• What Do Compressed Deep Neural Networks Forget?
• Dynamic Model Pruning with Feedback
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Measuring the Algorithmic Efficiency of Neural Networks
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
• Automatic Liver Lesion Detection using Cascaded Deep Residual Networks
• Regularizing CNNs with Locally Constrained Decorrelations
• An overview of mixing augmentation methods and augmentation strategies
• Adversarial Self-Supervised Contrastive Learning
• Discovering Parametric Activation Functions
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Deep Polynomial Neural Networks
• Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
• Heterogeneous Multilayer Generalized Operational Perceptron
• Interleaved Group Convolutions for Deep Neural Networks
• Mathematics of Deep Learning
• Nonlinear Approximation via Compositions
• Feature-map-level Online Adversarial Knowledge Distillation
• Sequential vessel segmentation via deep channel attention network
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• Scattering Networks for Hybrid Representation Learning
• Selfie: Self-supervised Pretraining for Image Embedding
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Attentive Recurrent Comparators
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• FreezeOut: Accelerate Training by Progressively Freezing Layers
• DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples
• Learning Sparse Networks Using Targeted Dropout
• Robust Pre-Training by Adversarial Contrastive Learning
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Gated-Dilated Networks for Lung Nodule Classification in CT scans
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Exploring the Space of Black-box Attacks on Deep Neural Networks
• Vision Transformer based COVID-19 Detection using Chest X-rays
• Cloud and Cloud Shadow Segmentation for Remote Sensing Imagery via Filtered Jaccard Loss Function and Parametric Augmentation
• NBDT: Neural-Backed Decision Trees
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• Residual Convolutional CTC Networks for Automatic Speech Recognition
• Selective Feature Connection Mechanism: Concatenating Multi-layer CNN Features with a Feature Selector
• Contrastive Representation Distillation
• SpaceNet: Make Free Space For Continual Learning
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments
• FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction
• ConTNet: Why not use convolution and transformer at the same time?
• BOIL: Towards Representation Change for Few-shot Learning
• Learning Feature Pyramids for Human Pose Estimation
• RCCNet: An Efficient Convolutional Neural Network for Histological Routine Colon Cancer Nuclei Classification
• Analysis and Optimization of Convolutional Neural Network Architectures
• i-RevNet: Deep Invertible Networks
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Detection of Face Recognition Adversarial Attacks
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Augmented Neural ODEs
• Multiscale Deep Equilibrium Models
• Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Deep Predictive Coding Network for Object Recognition
• The Early Phase of Neural Network Training
• Augment your batch: better training with larger batches
• A Constructive Prediction of the Generalization Error Across Scales
• Accurate Pulmonary Nodule Detection in Computed Tomography Images Using Deep Convolutional Neural Networks
• Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling
• Three Mechanisms of Weight Decay Regularization
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Deep Convolutional Neural Network Design Patterns
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Detecting Dementia from Speech and Transcripts using Transformers
• Structured Consistency Loss for semi-supervised semantic segmentation
• TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
• Data Valuation using Reinforcement Learning
• Overfitting in adversarially robust deep learning
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Probabilistic Binary Neural Networks
• Scale-Equivariant Steerable Networks
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation
• DeepMarks: A Digital Fingerprinting Framework for Deep Neural Networks
• Learning perturbation sets for robust machine learning
• BlackMarks: Blackbox Multibit Watermarking for Deep Neural Networks
• ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• Graph-based Knowledge Distillation by Multi-head Attention Network
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• More is Less: A More Complicated Network with Less Inference Complexity
• Class-Imbalanced Semi-Supervised Learning
• Comparative evaluation of CNN architectures for Image Caption Generation
• Unsupervised Outlier Detection using Memory and Contrastive Learning
• Accelerating Deep Learning by Focusing on the Biggest Losers
• Generalization Error of Invariant Classifiers
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
• Do Wider Neural Networks Really Help Adversarial Robustness?
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• Variational Information Distillation for Knowledge Transfer
• Scaling of neural-network quantum states for time evolution
• Defensive Quantization: When Efficiency Meets Robustness
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• GFF: Gated Fully Fusion for Semantic Segmentation
• Large batch size training of neural networks with adversarial training and second-order information
• A Deep Learning Approach for Pose Estimation from Volumetric OCT Data
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Deep Layer Aggregation
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Oriented Response Networks
• Towards Real-Time Multi-Object Tracking
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Stacked U-Nets: A No-Frills Approach to Natural Image Segmentation
• Leveraging the Feature Distribution in Transfer-based Few-Shot Learning
• Multi-Task Zipping via Layer-wise Neuron Sharing
• ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
• CryptoNAS: Private Inference on a ReLU Budget
• Geometry-aware Instance-reweighted Adversarial Training
• Tensor Normalization and Full Distribution Training
• Feature Fusion for Online Mutual Knowledge Distillation
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Contrastive Model Inversion for Data-Free Knowledge Distillation
• Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• Improving Auto-Augment via Augmentation-Wise Weight Sharing
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• Learning to Defend by Learning to Attack
• A Good Practice Towards Top Performance of Face Recognition: Transferred Deep Feature Fusion
• Deep Pyramidal Residual Networks
• Towards Understanding Label Smoothing
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• On Feature Normalization and Data Augmentation
• Assessing Post-Disaster Damage from Satellite Imagery using Semi-Supervised Learning Techniques
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Examples
• Wide Compression: Tensor Ring Nets
• Privado: Practical and Secure DNN Inference with Enclaves
• Self-Supervised Learning For Few-Shot Image Classification
• Scaling Laws for Deep Learning
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Automated Human Cell Classification in Sparse Datasets using Few-Shot Learning
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• Training DNNs with Hybrid Block Floating Point
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
• Continuous-in-Depth Neural Networks
• Neural Networks with Recurrent Generative Feedback
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• Robust Neural Networks using Randomized Adversarial Training
• S-Net: A Scalable Convolutional Neural Network for JPEG Compression Artifact Reduction
• Greedy Layerwise Learning Can Scale to ImageNet
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• GradAug: A New Regularization Method for Deep Neural Networks
• Consistent Estimators for Learning to Defer to an Expert
• Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
• Towards Understanding Fast Adversarial Training
• When Ensembling Smaller Models is More Efficient than Single Large Models
• DPGN: Distribution Propagation Graph Network for Few-shot Learning
• Rethinking Feature Distribution for Loss Functions in Image Classification
• DeepReDuce: ReLU Reduction for Fast Private Inference
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• FEED: Feature-level Ensemble for Knowledge Distillation
• Mitigating Bias in Calibration Error Estimation
• Test-time Batch Statistics Calibration for Covariate Shift
• ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networks
• Adaptive Regularization of Labels
• Provable Filter Pruning for Efficient Neural Networks
• Towards Compact and Robust Deep Neural Networks
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Hybrid Discriminative-Generative Training via Contrastive Learning
• End-to-End Multi-Task Learning with Attention
• Sill-Net: Feature Augmentation with Separated Illumination Representation
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Towards Feature Space Adversarial Attack
• Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors
• Lightweight Residual Densely Connected Convolutional Neural Network
• How Does Mixup Help With Robustness and Generalization?
• Practical Block-wise Neural Network Architecture Generation
• FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Learning Chained Deep Features and Classifiers for Cascade in Object Detection
• Reliable Adversarial Distillation with Unreliable Teachers
• SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systems
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Normalized Direction-preserving Adam
• Do deep nets really need weight decay and dropout?
• Parle: parallelizing stochastic gradient descent
• Scaling the Scattering Transform: Deep Hybrid Networks
• Differentiable Model Compression via Pseudo Quantization Noise
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• Improving Semantic Segmentation via Decoupled Body and Edge Supervision
• One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis
• A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning
• Finding Better Topologies for Deep Convolutional Neural Networks by Evolution
• Learning to Learn Variational Semantic Memory
• Towards an Adversarially Robust Normalization Approach
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Few-Shot Image Recognition by Predicting Parameters from Activations
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• HCM: Hardware-Aware Complexity Metric for Neural Network Architectures
• What causes the test error? Going beyond bias-variance via ANOVA
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Deep Learning based approach to detect Customer Age, Gender and Expression in Surveillance Video
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation
• Large Scale Visual Food Recognition
• Adversarial Training Versus Weight Decay
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
• PowerNorm: Rethinking Batch Normalization in Transformers
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity
• Deep learning for photoacoustic imaging: a survey
• MixNorm: Test-Time Adaptation Through Online Normalization Estimation
• Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
• Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets
• Deep Learning Assessment of galaxy morphology in S-PLUS DataRelease 1
• Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild
• Understanding the Interaction of Adversarial Training with Noisy Labels
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks
• Multi-scale Adaptive Task Attention Network for Few-Shot Learning
• Learning Neural Network Subspaces
• Deep Speech Enhancement for Reverberated and Noisy Signals using Wide Residual Networks
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• Recognizing Birds from Sound - The 2018 BirdCLEF Baseline System
• OVANet: One-vs-All Network for Universal Domain Adaptation
• Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
• Large Scale Private Learning via Low-rank Reparametrization
• Constructing Deep Neural Networks by Bayesian Network Structure Learning
• Generalization in Machine Learning via Analytical Learning Theory
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise
• Rethinking Bottleneck Structure for Efficient Mobile Network Design
• Progressive Feature Fusion Network for Realistic Image Dehazing
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• Convolution Neural Network Architecture Learning for Remote Sensing Scene Classification
• K-Hairstyle: A Large-scale Korean Hairstyle Dataset for Virtual Hair Editing and Hairstyle Classification
• C-RPNs: Promoting Object Detection in real world via a Cascade Structure of Region Proposal Networks
• Advancing System Performance with Redundancy: From Biological to Artificial Designs
• Densely Guided Knowledge Distillation using Multiple Teacher Assistants
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• Large Margin Deep Networks for Classification
• Active Image Synthesis for Efficient Labeling
• Two Souls in an Adversarial Image: Towards Universal Adversarial Example Detection using Multi-view Inconsistency
• Distributional Generalization: A New Kind of Generalization
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• Using Machine Learning at Scale in HPC Simulations with SmartSim: An Application to Ocean Climate Modeling
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• Active Convolution: Learning the Shape of Convolution for Image Classification
• Learning to play the Chess Variant Crazyhouse above World Champion Level with Deep Neural Networks and Human Data
• Trained Rank Pruning for Efficient Deep Neural Networks
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• Few-shot Classification via Adaptive Attention
• TestRank: Bringing Order into Unlabeled Test Instances for Deep Learning Tasks
• E-swish: Adjusting Activations to Different Network Depths
• LocalDrop: A Hybrid Regularization for Deep Neural Networks
• RIGA: Covert and Robust White-Box Watermarking of Deep Neural Networks
• An Enhanced Convolutional Neural Network in Side-Channel Attacks and Its Visualization
• Shaping representations through communication: community size effect in artificial learning systems
• WeMix: How to Better Utilize Data Augmentation
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• Test time Adaptation through Perturbation Robustness
• The Dual Information Bottleneck
• On the design of convolutional neural networks for automatic detection of Alzheimer's disease
• On the Relation between Color Image Denoising and Classification
• Time for a Background Check! Uncovering the impact of Background Features on Deep Neural Networks
• Multiresolution Convolutional Autoencoders
• DNN or k-NN: That is the Generalize vs. Memorize Question
• Learning data augmentation policies using augmented random search
• Deep Metric Transfer for Label Propagation with Limited Annotated Data
• Online Adversarial Purification based on Self-Supervision
• Domain-specific Communication Optimization for Distributed DNN Training
• Understanding Generalization in Deep Learning via Tensor Methods
• Convolutional Residual Memory Networks
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Unsupervised Deep Features for Remote Sensing Image Matching via Discriminator Network
• Diversity inducing Information Bottleneck in Model Ensembles
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Low-Precision Batch-Normalized Activations
• ATHENA: A Framework based on Diverse Weak Defenses for Building Adversarial Defense
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• On Improving Adversarial Transferability of Vision Transformers
• Bridging the Performance Gap between FGSM and PGD Adversarial Training
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets
• Full deep neural network training on a pruned weight budget
• Sparse Weight Activation Training
• Rethinking FUN: Frequency-Domain Utilization Networks
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Rethinking and Improving the Robustness of Image Style Transfer
• Cyclic Differentiable Architecture Search
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks
• Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs
• Efficient Adversarial Training with Transferable Adversarial Examples
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Batch Group Normalization
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• Neural Architecture Dilation for Adversarial Robustness
• Anomalous Example Detection in Deep Learning: A Survey
• On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder
• Adversarial Token Attacks on Vision Transformers
• Adversarial Robustness Assessment: Why both $L_0$ and $L_{\infty }$ Attacks Are Necessary
• Intra-model Variability in COVID-19 Classification Using Chest X-ray Images
• Should Ensemble Members Be Calibrated?
• A Gap-Based Framework for Chinese Word Segmentation via Very Deep Convolutional Networks
• Online Hyper-parameter Learning for Auto-Augmentation Strategy
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Combining Ensembles and Data Augmentation can Harm your Calibration
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
• Adversarially Adaptive Normalization for Single Domain Generalization
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms
• Transfer learning based few-shot classification using optimal transport mapping from preprocessed latent space of backbone neural network
• Convolutional Networks with Dense Connectivity
• Robust or Private? Adversarial Training Makes Models More Vulnerable to Privacy Attacks
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• Query Attack via Opposite-Direction Feature:Towards Robust Image Retrieval
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• Compact Global Descriptor for Neural Networks
• Adversarial Visual Robustness by Causal Intervention
• Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation
• Learning Better Internal Structure of Words for Sequence Labeling
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• Deep CNNs for Peripheral Blood Cell Classification
• Orthogonal Deep Neural Networks
• Boosting Black-Box Attack with Partially Transferred Conditional Adversarial Distribution
• On the asymptotics of wide networks with polynomial activations
• Graph-based Interpolation of Feature Vectors for Accurate Few-Shot Classification
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• A Critical Evaluation of Open-World Machine Learning
• Is Support Set Diversity Necessary for Meta-Learning?
• IRLAS: Inverse Reinforcement Learning for Architecture Search
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• Weakly Supervised Clustering by Exploiting Unique Class Count
• Faster Neural Network Training with Approximate Tensor Operations
• A Generic Network Compression Framework for Sequential Recommender Systems
• Automated Learning Rate Scheduler for Large-batch Training
• Deep Transform and Metric Learning Network: Wedding Deep Dictionary Learning and Neural Networks
• I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively
• Coresets for Robust Training of Neural Networks against Noisy Labels
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• CIFAR-10: KNN-based Ensemble of Classifiers
• Random Noise Defense Against Query-Based Black-Box Attacks
• Implicit Rugosity Regularization via Data Augmentation
• On the interplay between data structure and loss function in classification problems
• At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
• RayS: A Ray Searching Method for Hard-label Adversarial Attack
• Real-time Human Pose Estimation from Video with Convolutional Neural Networks
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Improving Query Efficiency of Black-box Adversarial Attack
• Detecting Overfitting via Adversarial Examples
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• For self-supervised learning, Rationality implies generalization, provably
• Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images
• Towards Learning Affine-Invariant Representations via Data-Efficient CNNs
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
• EEGdenoiseNet: A benchmark dataset for end-to-end deep learning solutions of EEG denoising
• The State of Knowledge Distillation for Classification
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• TinyAction Challenge: Recognizing Real-world Low-resolution Activities in Videos
• Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
• Parallel Grid Pooling for Data Augmentation
• A Survey on Fault-tolerance in Distributed Optimization and Machine Learning
• Convolution Aware Initialization
• Guided Interpolation for Adversarial Training
• Rethinking the Number of Channels for the Convolutional Neural Network
• DeepOBS: A Deep Learning Optimizer Benchmark Suite
• Learnable Descent Algorithm for Nonsmooth Nonconvex Image Reconstruction
• Listen carefully and tell: an audio captioning system based on residual learning and gammatone audio representation
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• Enhancing Adversarial Defense by k-Winners-Take-All
• Trade-offs and Guarantees of Adversarial Representation Learning for Information Obfuscation
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
• ResNet After All? Neural ODEs and Their Numerical Solution
• A Loss Curvature Perspective on Training Instability in Deep Learning
• SOAR: Second-Order Adversarial Regularization
• Generalized Adaptation for Few-Shot Learning
• Harmonic Networks: Integrating Spectral Information into CNNs
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Simulating Personal Food Consumption Patterns using a Modified Markov Chain
• Robust and On-the-fly Dataset Denoising for Image Classification
• Reversed Active Learning based Atrous DenseNet for Pathological Image Classification
• On Detecting GANs and Retouching based Synthetic Alterations
• Supervised COSMOS Autoencoder: Learning Beyond the Euclidean Loss!
• Soft Labeling Affects Out-of-Distribution Detection of Deep Neural Networks
• An Experience-based Direct Generation approach to Automatic Image Cropping
• Zero and Few Shot Learning with Semantic Feature Synthesis and Competitive Learning
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• Residual Convolutional Neural Network Revisited with Active Weighted Mapping
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Explainable Deep Learning for Uncovering Actionable Scientific Insights for Materials Discovery and Design
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• Regional Adversarial Training for Better Robust Generalization
• How and When Adversarial Robustness Transfers in Knowledge Distillation?
• DeepPeep: Exploiting Design Ramifications to Decipher the Architecture of Compact DNNs
• Rethink ReLU to Training Better CNNs
• Beyond Forward Shortcuts: Fully Convolutional Master-Slave Networks (MSNets) with Backward Skip Connections for Semantic Segmentation
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device
• Multi-head Knowledge Distillation for Model Compression
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• Generalizing Deep Models for Overhead Image Segmentation Through Getis-Ord Gi* Pooling
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• MetaDelta: A Meta-Learning System for Few-shot Image Classification
• Pufferfish: Communication-efficient Models At No Extra Cost
• Why bigger is not always better: on finite and infinite neural networks
• Negative sampling in semi-supervised learning
• Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks
• Towards Oracle Knowledge Distillation with Neural Architecture Search
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• Chameleon: Learning Model Initializations Across Tasks With Different Schemas
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• SORT: Second-Order Response Transform for Visual Recognition
• Intra-Ensemble in Neural Networks
• Switching Transferable Gradient Directions for Query-Efficient Black-Box Adversarial Attacks
• Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms
• One-bit Supervision for Image Classification
• A Study of Checkpointing in Large Scale Training of Deep Neural Networks
• Supervised Deep Neural Networks (DNNs) for Pricing/Calibration of Vanilla/Exotic Options Under Various Different Processes
• Simplified Stochastic Feedforward Neural Networks
• QEBA: Query-Efficient Boundary-Based Blackbox Attack
• Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
• GBCNs: Genetic Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• Semi-Supervised and Active Few-Shot Learning with Prototypical Networks
• Collage Inference: Using Coded Redundancy for Low Variance Distributed Image Classification
• Does Data Augmentation Benefit from Split BatchNorms
• Learning Representations that Support Robust Transfer of Predictors
• Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• Lifelong Learning with Sketched Structural Regularization
• Dynamic Sparse Graph for Efficient Deep Learning
• RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss
• DeepNNK: Explaining deep models and their generalization using polytope interpolation
• Functional Gradient Boosting based on Residual Network Perception
• 3D Aggregated Faster R-CNN for General Lesion Detection
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
• Online Knowledge Distillation with Diverse Peers
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• $\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
• Data Quality Matters For Adversarial Training: An Empirical Study
• Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
• Knowledge distillation for optimization of quantized deep neural networks
• On Calibration of Mixup Training for Deep Neural Networks
• Learning Optimal Data Augmentation Policies via Bayesian Optimization for Image Classification Tasks
• Curriculum By Smoothing
• DAAS: Differentiable Architecture and Augmentation Policy Search
• A New Look at Ghost Normalization
• EDAS: Efficient and Differentiable Architecture Search
• Toward Adversarial Robustness via Semi-supervised Robust Training
• Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections
• VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• Context-aware PolyUNet for Liver and Lesion Segmentation from Abdominal CT Images
• Understanding the Impact of Label Granularity on CNN-based Image Classification
• Lottery Jackpots Exist in Pre-trained Models
• ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets
• Overcoming Catastrophic Forgetting by Generative Regularization
• Few-Shot Learning by Integrating Spatial and Frequency Representation
• A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
• Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification
• Neural Network Encapsulation
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• Calibration of Neural Networks using Splines
• Factorized Bilinear Models for Image Recognition
• Deep Dimension Reduction for Supervised Representation Learning
• Energy-efficient Amortized Inference with Cascaded Deep Classifiers
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• OnlineAugment: Online Data Augmentation with Less Domain Knowledge
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
• Robust Visual Object Tracking with Two-Stream Residual Convolutional Networks
• Improved Adversarial Training via Learned Optimizer
• Student Network Learning via Evolutionary Knowledge Distillation
• Wavelet Denoised-ResNet CNN and LightGBM Method to Predict Forex Rate of Change
• Robustness, Privacy, and Generalization of Adversarial Training
• Implicit bias of deep linear networks in the large learning rate phase
• StackMix: A complementary Mix algorithm
• Parameter Prediction for Unseen Deep Architectures
• Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• Adaptive Feature Alignment for Adversarial Training
• Learnable Adaptive Cosine Estimator (LACE) for Image Classification
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Full-attention based Neural Architecture Search using Context Auto-regression
• Improving Model Robustness with Latent Distribution Locally and Globally
• Contextual Classification Using Self-Supervised Auxiliary Models for Deep Neural Networks
• Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
• Improve Adversarial Robustness via Weight Penalization on Classification Layer
• How Out-of-Distribution Data Hurts Semi-Supervised Learning
• Heuristic Rank Selection with Progressively Searching Tensor Ring Network
• Representation Transfer by Optimal Transport
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
• Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics
• Training convolutional neural networks with cheap convolutions and online distillation
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Putting visual object recognition in context
• Binarized Neural Architecture Search
• Decoder Choice Network for Meta-Learning
• Adversarial Transformations for Semi-Supervised Learning
• NLNL: Negative Learning for Noisy Labels
• ResNetX: a more disordered and deeper network architecture
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Weighting Is Worth the Wait: Bayesian Optimization with Importance Sampling
• Leveraging Class Similarity to Improve Deep Neural Network Robustness
• MDCN: Multi-Scale, Deep Inception Convolutional Neural Networks for Efficient Object Detection
• Backpropagation for Implicit Spectral Densities
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Classification of simulated radio signals using Wide Residual Networks for use in the search for extra-terrestrial intelligence
• Building a Regular Decision Boundary with Deep Networks
• Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks
• Non-Parametric Transformation Networks
• Gradually Updated Neural Networks for Large-Scale Image Recognition
• TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References
• Learning Less-Overlapping Representations
• Large-Scale Optimal Transport via Adversarial Training with Cycle-Consistency
• Understanding and Diagnosing Vulnerability under Adversarial Attacks
• Metric learning by Similarity Network for Deep Semi-Supervised Learning
• A Novel Learnable Gradient Descent Type Algorithm for Non-convex Non-smooth Inverse Problems
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• A study on the role of subsidiary information in replay attack spoofing detection
• Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss
• Confidence Calibration with Bounded Error Using Transformations
• SuperNet -- An efficient method of neural networks ensembling
• Medical Knowledge-Guided Deep Learning for Imbalanced Medical Image Classification
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• Deep Learning in Memristive Nanowire Networks
• Improving Differentially Private Models with Active Learning
• Actionable Attribution Maps for Scientific Machine Learning
• A Unified Game-Theoretic Interpretation of Adversarial Robustness
• Compressive Visual Representations
• Truncating Wide Networks using Binary Tree Architectures
• Knowledge Distillation Meets Self-Supervision
• Smoothness Analysis of Adversarial Training
• CrescendoNet: A Simple Deep Convolutional Neural Network with Ensemble Behavior
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Selective sampling for accelerating training of deep neural networks
• GM-Net: Learning Features with More Efficiency
• Random Bundle: Brain Metastases Segmentation Ensembling through Annotation Randomization
• Perceptually Constrained Adversarial Attacks
• Ensemble Transfer Learning for Emergency Landing Field Identification on Moderate Resource Heterogeneous Kubernetes Cluster
• Exploring Model Robustness with Adaptive Networks and Improved Adversarial Training
• Regularized Evolutionary Population-Based Training
• Data augmentation with Mobius transformations
• Uniform Priors for Data-Efficient Transfer
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Margin-Based Regularization and Selective Sampling in Deep Neural Networks
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
• An Attention Module for Convolutional Neural Networks
• Progressive Representative Labeling for Deep Semi-Supervised Learning
• Attention-based fusion of semantic boundary and non-boundary information to improve semantic segmentation
• Follow Your Path: a Progressive Method for Knowledge Distillation
• Pseudo-Representation Labeling Semi-Supervised Learning
• On Out-of-distribution Detection with Energy-based Models
• On Norm-Agnostic Robustness of Adversarial Training
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Learned Proximal Networks for Quantitative Susceptibility Mapping
• Defective Convolutional Networks
• Approximated Orthonormal Normalisation in Training Neural Networks
• Deep Neural Network Ensembles
• Utilizing Network Properties to Detect Erroneous Inputs
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Image Restoration Using Deep Regulated Convolutional Networks
• Multi-objective Search of Robust Neural Architectures against Multiple Types of Adversarial Attacks
• Supervised Deep Sparse Coding Networks
• Fused Deep Convolutional Neural Network for Precision Diagnosis of COVID-19 Using Chest X-Ray Images
• How Convolutional Neural Network Architecture Biases Learned Opponency and Colour Tuning
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• Feature Distillation With Guided Adversarial Contrastive Learning
• Gradient-based Data Augmentation for Semi-Supervised Learning
• SIPA: A Simple Framework for Efficient Networks
• Complementary Relation Contrastive Distillation
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Effective Regularization Through Loss-Function Metalearning
• Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
• Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
• Matching Distributions via Optimal Transport for Semi-Supervised Learning
• Content-Adaptive Pixel Discretization to Improve Model Robustness
• Augmentation Inside the Network
• Removing Undesirable Feature Contributions Using Out-of-Distribution Data
• ReRankMatch: Semi-Supervised Learning with Semantics-Oriented Similarity Representation
• Sequential Random Network for Fine-grained Image Classification
• Local Patch AutoAugment with Multi-Agent Collaboration
• Radius-margin bounds for deep neural networks
• Domain Invariant Adversarial Learning
• CRL: Class Representative Learning for Image Classification
• Adaptive Clustering of Robust Semantic Representations for Adversarial Image Purification
• Integrating Multiple Receptive Fields through Grouped Active Convolution
• Dynamic Defense Approach for Adversarial Robustness in Deep Neural Networks via Stochastic Ensemble Smoothed Model
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Greedy Network Enlarging
• Deep Collaborative Learning for Visual Recognition
• Prune Your Model Before Distill It
• An Exploration of Mimic Architectures for Residual Network Based Spectral Mapping
• FALCON: Feature Driven Selective Classification for Energy-Efficient Image Recognition
• FOCUS: Familiar Objects in Common and Uncommon Settings
• Parallel Deep Neural Networks Have Zero Duality Gap
• Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers
• ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees
• Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization
• What augmentations are sensitive to hyper-parameters and why?
• SPI-Optimizer: an integral-Separated PI Controller for Stochastic Optimization
• On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications
• CircConv: A Structured Convolution with Low Complexity
• Temporally Resolution Decrement: Utilizing the Shape Consistency for Higher Computational Efficiency
• Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
• GAN-based Pose-aware Regulation for Video-based Person Re-identification
• AI Driven Road Maintenance Inspection
• A Low-Compexity Deep Learning Framework For Acoustic Scene Classification
• ESFNet: Efficient Network for Building Extraction from High-Resolution Aerial Images
• Photozilla: A Large-Scale Photography Dataset and Visual Embedding for 20 Photography Styles
• Using Anomaly Feature Vectors for Detecting, Classifying and Warning of Outlier Adversarial Examples
• On The Distribution of Penultimate Activations of Classification Networks
• Learning a Discriminant Latent Space with Neural Discriminant Analysis
• DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Data Summarization via Bilevel Optimization
• GCCN: Global Context Convolutional Network
• Signature-Graph Networks
• D-PCN: Parallel Convolutional Networks for Image Recognition via a Discriminator
• LogAvgExp Provides a Principled and Performant Global Pooling Operator
• Scalable Bayesian neural networks by layer-wise input augmentation
• Data Augmentation via Structured Adversarial Perturbations
• Deep Asymmetric Multi-task Feature Learning
• An Autonomous Approach to Measure Social Distances and Hygienic Practices during COVID-19 Pandemic in Public Open Spaces
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Convergence of backpropagation with momentum for network architectures with skip connections
• Multi-level Feature Fusion-based CNN for Local Climate Zone Classification from Sentinel-2 Images: Benchmark Results on the So2Sat LCZ42 Dataset
• Contrastive Weight Regularization for Large Minibatch SGD
• Boosting the Performance of Semi-Supervised Learning with Unsupervised Clustering
• Out-Of-Distribution Detection With Subspace Techniques And Probabilistic Modeling Of Features
• MetaAugment: Sample-Aware Data Augmentation Policy Learning
• Road images augmentation with synthetic traffic signs using neural networks
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Classification of Long Noncoding RNA Elements Using Deep Convolutional Neural Networks and Siamese Networks
• PSDNet and DPDNet: Efficient channel expansion, Depthwise-Pointwise-Depthwise Inverted Bottleneck Block
• Knowledge Transfer Graph for Deep Collaborative Learning
• Matching the Clinical Reality: Accurate OCT-Based Diagnosis From Few Labels
• Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Fast Jacobian-Vector Product for Deep Networks
• Stochastic Whitening Batch Normalization
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Learning to Collaborate for User-Controlled Privacy
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Self-supervision of Feature Transformation for Further Improving Supervised Learning
• End-to-End Bengali Speech Recognition
• RotationOut as a Regularization Method for Neural Network
• Go Small and Similar: A Simple Output Decay Brings Better Performance
• Deep Virtual Networks for Memory Efficient Inference of Multiple Tasks
• Graphs for deep learning representations
• A Simple Riemannian Manifold Network for Image Set Classification
• ShuffleBlock: Shuffle to Regularize Deep Convolutional Neural Networks
• Generalization by Recognizing Confusion
• Deep Anchored Convolutional Neural Networks
• Multi-Scale Spatially-Asymmetric Recalibration for Image Classification
• PR Product: A Substitute for Inner Product in Neural Networks
• Constraining Logits by Bounded Function for Adversarial Robustness
• SegCodeNet: Color-Coded Segmentation Masks for Activity Detection from Wearable Cameras
• Built-in Elastic Transformations for Improved Robustness
• Mitigating deep double descent by concatenating inputs
• Spatially Attentive Output Layer for Image Classification
• PERMDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matrices
• Improving Visual Recognition using Ambient Sound for Supervision
• Training robust anomaly detection using ML-Enhanced simulations
• Light Multi-segment Activation for Model Compression
• Exponential Discriminative Metric Embedding in Deep Learning
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
• Privacy Preserving Recalibration under Domain Shift
• Learning to Reweight with Deep Interactions
• State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations
• Using an ensemble color space model to tackle adversarial examples
• Langevin Cooling for Domain Translation
• Batch Normalization and the impact of batch structure on the behavior of deep convolution networks
• Learning Multiple Categories on Deep Convolution Networks
• Online Filter Clustering and Pruning for Efficient Convnets
• Unsupervised and Supervised Structure Learning for Protein Contact Prediction
• Convolutional Hashing for Automated Scene Matching
• Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks
• Super Interaction Neural Network
• Exploiting Nontrivial Connectivity for Automatic Speech Recognition
• Robust Temporal Ensembling for Learning with Noisy Labels
• Hierarchical Auxiliary Learning
• Associative Convolutional Layers
• Max and Coincidence Neurons in Neural Networks
• Efficient Modelling Across Time of Human Actions and Interactions
• Calibrated Adversarial Training
• Orientation Convolutional Networks for Image Recognition
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
• clcNet: Improving the Efficiency of Convolutional Neural Network using Channel Local Convolutions
• Mixture separability loss in a deep convolutional network for image classification
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
• Privacy-Aware Activity Classification from First Person Office Videos
• Multi-Pretext Attention Network for Few-shot Learning with Self-supervision
• Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task
• Self-Denoising Neural Networks for Few Shot Learning
• Predictive Geological Mapping with Convolution Neural Network Using Statistical Data Augmentation on a 3D Model
• Enhancing Transformation-based Defenses using a Distribution Classifier
• Multi-way Encoding for Robustness
• Differentiable Combinatorial Losses through Generalized Gradients of Linear Programs
• Strategies for Robust Image Classification
• Architectural Resilience to Foreground-and-Background Adversarial Noise
• Deep Goal-Oriented Clustering
• "You might also like this model": Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets
• Skeptical Deep Learning with Distribution Correction
• FILTRA: Rethinking Steerable CNN by Filter Transform
• Semantics-Preserving Adversarial Training
• KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation
• Sequenced-Replacement Sampling for Deep Learning
• Neural Optimization Kernel: Towards Robust Deep Learning
• What can linear interpolation of neural network loss landscapes tell us?
• Optimizing Data Augmentation Policy Through Random Unidimensional Search
• Learning Inward Scaled Hypersphere Embedding: Exploring Projections in Higher Dimensions
• CloudifierNet -- Deep Vision Models for Artificial Image Processing
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Understanding Classifier Mistakes with Generative Models
• Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets
• Learning from Matured Dumb Teacher for Fine Generalization
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions
• Learning with Hyperspherical Uniformity
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• Sub-clusters of Normal Data for Anomaly Detection
• Gabor filter incorporated CNN for compression
• ConAM: Confidence Attention Module for Convolutional Neural Networks
• Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout
• Relative Depth Order Estimation Using Multi-scale Densely Connected Convolutional Networks
• Comb Convolution for Efficient Convolutional Architecture
• Modulated Self-attention Convolutional Network for VQA
• Constant Random Perturbations Provide Adversarial Robustness with Minimal Effect on Accuracy
• Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation
• Dual Head Adversarial Training
• Self-Gradient Networks
• Deep Competitive Pathway Networks
• Deep Adaptive Wavelet Network
• Spectral Tensor Train Parameterization of Deep Learning Layers
• SwGridNet: A Deep Convolutional Neural Network based on Grid Topology for Image Classification
• Boosted CVaR Classification
• Unbounded Output Networks for Classification
• Attend and Rectify: a Gated Attention Mechanism for Fine-Grained Recovery
• Noisy Feature Mixup
• Rethinking Uncertainty in Deep Learning: Whether and How it Improves Robustness
• Variance Reduction in Deep Learning: More Momentum is All You Need
• ChebLieNet: Invariant Spectral Graph NNs Turned Equivariant by Riemannian Geometry on Lie Groups
• SelectScale: Mining More Patterns from Images via Selective and Soft Dropout
• Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization
• Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
• Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embedding
• Customer Analytics using Surveillance Video
• Deep Feature Pyramid Convolutional Networks with In-Place Activated Batch Normalization for Automated Skin Lesion Boundary Segmentation
• Bayesian Optimized 1-Bit CNNs
• Multiclass non-Adversarial Image Synthesis, with Application to Classification from Very Small Sample
• Computing Class Hierarchies from Classifiers