On Calibration of Modern Neural Networks

Abstract

Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

arXiv commentICML 2017

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Achieving Human Parity in Conversational Speech Recognition
• Deep Networks with Stochastic Depth
• Learning Scalable Deep Kernels with Recurrent Structure
• Deep Kernel Learning

Cited by in corpus (583)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Skillful Precipitation Nowcasting using Deep Generative Models of Radar
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• When Does Label Smoothing Help?
• Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• Global canopy height regression and uncertainty estimation from GEDI LIDAR waveforms with deep ensembles
• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Measuring Massive Multitask Language Understanding
• To Trust Or Not To Trust A Classifier
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Pathologies of Neural Models Make Interpretations Difficult
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Stochastic Activation Pruning for Robust Adversarial Defense
• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis
• Natural Adversarial Examples
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Learned Cardinalities: Estimating Correlated Joins with Deep Learning
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Measuring Calibration in Deep Learning
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Earthquake magnitude and location estimation from real time seismic waveforms with a transformer network
• Practical Deep Learning with Bayesian Principles
• Scalable agent alignment via reward modeling: a research direction
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• RobustBench: a standardized adversarial robustness benchmark
• AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild
• Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• Multivariate Confidence Calibration for Object Detection
• Active label cleaning for improved dataset quality under resource constraints
• Exploring the Limits of Out-of-Distribution Detection
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• What Are Bayesian Neural Network Posteriors Really Like?
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
• Bayesian deep learning for error estimation in the analysis of anomalous diffusion
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Unsolved Problems in ML Safety
• What Do Compressed Deep Neural Networks Forget?
• Does deep learning model calibration improve performance in class-imbalanced medical image classification?
• Verified Uncertainty Calibration
• The Pitfalls of Simplicity Bias in Neural Networks
• Calibrating Deep Neural Networks using Focal Loss
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier that Validates 301 New Exoplanets
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• A Review of Testing Object-Based Environment Perception for Safe Automated Driving
• Unrestricted Adversarial Examples
• Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Implicit Weight Uncertainty in Neural Networks
• Cycle Self-Training for Domain Adaptation
• Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels
• Predicting the hosts of prokaryotic viruses using GCN-based semi-supervised learning
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Learning phase transitions from dynamics
• Temporal Stability in Predictive Process Monitoring
• Evidential fully convolutional network for semantic segmentation
• Analyzing Uncertainty in Neural Machine Translation
• Salient Objects in Clutter
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch
• Self-Distillation as Instance-Specific Label Smoothing
• Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks
• Long-tail learning via logit adjustment
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics
• Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination
• Distribution Matching in Variational Inference
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Denoising Multi-Source Weak Supervision for Neural Text Classification
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning
• Detection of Gravitational Waves Using Bayesian Neural Networks
• A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems
• Discriminative out-of-distribution detection for semantic segmentation
• Combination of Multiple Global Descriptors for Image Retrieval
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation
• Deep Weighted Averaging Classifiers
• A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
• Dynamic Neural Networks: A Survey
• A Post-processing Method for Detecting Unknown Intent of Dialogue System via Pre-trained Deep Neural Network Classifier
• Danish Fungi 2020 -- Not Just Another Image Recognition Dataset
• Task adapted reconstruction for inverse problems
• Unsupervised Domain Adaptation of Black-Box Source Models
• Explicit Regularisation in Gaussian Noise Injections
• Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images
• Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
• Simple and Effective VAE Training with Calibrated Decoders
• PAD: Towards Principled Adversarial Malware Detection Against Evasion Attacks
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Confidence Regularized Self-Training
• The Unreasonable Effectiveness of Deep Evidential Regression
• Uncertainty-Aware Attention for Reliable Interpretation and Prediction
• Design by adaptive sampling
• Uncertainty Sets for Image Classifiers using Conformal Prediction
• Uncertainty Quantification and Deep Ensembles
• Robust Out-of-distribution Detection for Neural Networks
• CaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows
• Information Aware Max-Norm Dirichlet Networks for Predictive Uncertainty Estimation
• Adversarial Distillation of Bayesian Neural Network Posteriors
• Dynamic Decision Tree Ensembles for Energy-Efficient Inference on IoT Edge Nodes
• SOS: Selective Objective Switch for Rapid Immunofluorescence Whole Slide Image Classification
• Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction
• Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples
• Meta-Learned Confidence for Few-shot Learning
• Prediction Calibration for Generalized Few-shot Semantic Segmentation
• Monitoring and explainability of models in production
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
• Outside the Box: Abstraction-Based Monitoring of Neural Networks
• Trustable Co-label Learning from Multiple Noisy Annotators
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Into the Unknown: Active Monitoring of Neural Networks
• Adversarial Phenomenon in the Eyes of Bayesian Deep Learning
• On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
• Laplace Redux -- Effortless Bayesian Deep Learning
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• AI Research Considerations for Human Existential Safety (ARCHES)
• Multi-Scored Sleep Databases: How to Exploit the Multiple-Labels in Automated Sleep Scoring
• Uncertainty-Aware Reliable Text Classification
• LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
• SphereFace2: Binary Classification is All You Need for Deep Face Recognition
• Noisy Natural Gradient as Variational Inference
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Minimum Class Confusion for Versatile Domain Adaptation
• Demonstration of background rejection using deep convolutional neural networks in the NEXT experiment
• CaloFlow for CaloChallenge Dataset 1
• Consistent Estimators for Learning to Defer to an Expert
• Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference
• Efficient Bayesian Uncertainty Estimation for nnU-Net
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization
• Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference
• Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data
• Toward Degree Bias in Embedding-Based Knowledge Graph Completion
• Programming and Training Rate-Independent Chemical Reaction Networks
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection
• Mitigating Bias in Calibration Error Estimation
• Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training
• Countdown Regression: Sharp and Calibrated Survival Predictions
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• A Less Biased Evaluation of Out-of-distribution Sample Detectors
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• The Loss Surfaces of Neural Networks with General Activation Functions
• Rate Distortion For Model Compression: From Theory To Practice
• Assessing Generalization of SGD via Disagreement
• Transferable Calibration with Lower Bias and Variance in Domain Adaptation
• Automatic Large Scale Detection of Red Palm Weevil Infestation using Aerial and Street View Images
• Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Uncertainty-Aware Machine Translation Evaluation
• A Bayesian Approach to In-Game Win Probability in Soccer
• Learning from Positive and Unlabeled Data with Arbitrary Positive Shift
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Calibrating Uncertainties in Object Localization Task
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation
• Improving Simple Models with Confidence Profiles
• xGEMs: Generating Examplars to Explain Black-Box Models
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Can We Leverage Predictive Uncertainty to Detect Dataset Shift and Adversarial Examples in Android Malware Detection?
• Multimodal Detection of Unknown Objects on Roads for Autonomous Driving
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Towards Cross-Granularity Few-Shot Learning: Coarse-to-Fine Pseudo-Labeling with Visual-Semantic Meta-Embedding
• How Much Can We Really Trust You? Towards Simple, Interpretable Trust Quantification Metrics for Deep Neural Networks
• Complementary-Label Learning for Arbitrary Losses and Models
• One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction
• Implicit competitive regularization in GANs
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• On Episodes, Prototypical Networks, and Few-shot Learning
• Learning Neural Network Subspaces
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Survival Function Matching for Calibrated Time-to-Event Predictions
• ProtoAttend: Attention-Based Prototypical Learning
• Calibrating Deep Convolutional Gaussian Processes
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• Set-valued classification -- overview via a unified framework
• ExplainaBoard: An Explainable Leaderboard for NLP
• Early Rumor Detection Using Neural Hawkes Process with a New Benchmark Dataset
• Distributional Generalization: A New Kind of Generalization
• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• Reliable training and estimation of variance networks
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning
• Learning Intra-Batch Connections for Deep Metric Learning
• Analysis of the first Genetic Engineering Attribution Challenge
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving
• DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
• Diverse Ensembles Improve Calibration
• Selective prediction-set models with coverage guarantees
• Does Knowledge Distillation Really Work?
• Bayesian Deep Learning via Subnetwork Inference
• Machine Learning for Cataract Classification and Grading on Ophthalmic Imaging Modalities: A Survey
• Scalable Gaussian Processes on Discrete Domains
• Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation
• Acoustic scene classification using teacher-student learning with soft-labels
• PredDiff: Explanations and Interactions from Conditional Expectations
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Sustainable Edge Intelligence Through Energy-Aware Early Exiting
• Learning Strategies in Decentralized Matching Markets under Uncertain Preferences
• Predictive Uncertainty Quantification with Compound Density Networks
• Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streams
• Learning Optimal Conformal Classifiers
• Quantifying Deep Learning Model Uncertainty in Conformal Prediction
• Detecting, Localising and Classifying Polyps from Colonoscopy Videos using Deep Learning
• Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs
• Improving Video Instance Segmentation by Light-weight Temporal Uncertainty Estimates
• Online Adaptation to Label Distribution Shift
• On Calibration and Out-of-domain Generalization
• Towards Automating the AI Operations Lifecycle
• A Unified View of Label Shift Estimation
• Label-Imbalanced and Group-Sensitive Classification under Overparameterization
• Angular Visual Hardness
• Being Bayesian about Categorical Probability
• Channel-driven Decentralized Bayesian Federated Learning for Trustworthy Decision Making in D2D Networks
• Unsupervised Domain Adaptation in the Absence of Source Data
• Should Ensemble Members Be Calibrated?
• ReMix: Calibrated Resampling for Class Imbalance in Deep learning
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Assurance Monitoring of Learning Enabled Cyber-Physical Systems Using Inductive Conformal Prediction based on Distance Learning
• Early-stopped neural networks are consistent
• Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusion
• Towards reliable calcification detection: calibration of uncertainty in coronary optical coherence tomography images
• Confidence-Calibrated Face and Kinship Verification
• Certifying Confidence via Randomized Smoothing
• Combining Ensembles and Data Augmentation can Harm your Calibration
• On Mixup Regularization
• Distribution-free binary classification: prediction sets, confidence intervals and calibration
• Overinterpretation reveals image classification model pathologies
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Assurance Monitoring of Cyber-Physical Systems with Machine Learning Components
• Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off
• Supermasks in Superposition
• When and How Mixup Improves Calibration
• Classification with Rejection Based on Cost-sensitive Classification
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
• Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes
• Interpretable Uncertainty Quantification in AI for HEP
• Likelihood Ratios for Out-of-Distribution Detection
• Controlling Risk of Web Question Answering
• DeepLens: Interactive Out-of-distribution Data Detection in NLP Models
• Leveraging Angular Distributions for Improved Knowledge Distillation
• Change-point detection in anomalous-diffusion trajectories utilising machine-learning-based uncertainty estimates
• Out-of-domain Detection for Natural Language Understanding in Dialog Systems
• Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection
• Adaptive Label Smoothing
• Hierarchical Classification of Pulmonary Lesions: A Large-Scale Radio-Pathomics Study
• On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration
• Soft Calibration Objectives for Neural Networks
• Domain Adaptation with Auxiliary Target Domain-Oriented Classifier
• Using Multiple Dermoscopic Photographs of One Lesion Improves Melanoma Classification via Deep Learning: A Prognostic Diagnostic Accuracy Study
• On the Convergence and Calibration of Deep Learning with Differential Privacy
• Diversity regularization in deep ensembles
• Detecting OODs as datapoints with High Uncertainty
• Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector
• On the Role of Dataset Quality and Heterogeneity in Model Confidence
• Semi-Supervised Learning with Normalizing Flows
• Progressive Neural Networks for Image Classification
• Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
• Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
• On the use of uncertainty in classifying Aedes Albopictus mosquitoes
• Label-Noise Robust Generative Adversarial Networks
• Quizbowl: The Case for Incremental Question Answering
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks
• Probing the Purview of Neural Networks via Gradient Analysis
• To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictions
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
• Augmenting GAIL with BC for sample efficient imitation learning
• A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticity
• Optimising HEP parameter fits via Monte Carlo weight derivative regression
• Automatically detecting data drift in machine learning classifiers
• Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters
• Regularizing Neural Networks via Adversarial Model Perturbation
• Distribution-free uncertainty quantification for classification under label shift
• Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification
• On the convergence of PINNs
• Decoding Neural Activity to Assess Individual Latent State in Ecologically Valid Contexts
• Detecting Out-of-distribution Samples via Variational Auto-encoder with Reliable Uncertainty Estimation
• KS(conf ): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications
• Distribution-free calibration guarantees for histogram binning without sample splitting
• Improved GQ-CNN: Deep Learning Model for Planning Robust Grasps
• Vision-Based Uncertainty-Aware Motion Planning based on Probabilistic Semantic Segmentation
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Improving Robustness to Model Inversion Attacks via Mutual Information Regularization
• Interpreting Neural Networks Using Flip Points
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
• Dynamic Scale Inference by Entropy Minimization
• The Fairness-Accuracy Pareto Front
• The Battleship Approach to the Low Resource Entity Matching Problem
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• Fast and Accurate Factual Inconsistency Detection Over Long Documents
• Cold-start Active Learning through Self-supervised Language Modeling
• Fast and Reliable Probabilistic Face Embeddings in the Wild
• PAC Confidence Predictions for Deep Neural Network Classifiers
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• Consistent Sparse Deep Learning: Theory and Computation
• Explaining Predictions by Approximating the Local Decision Boundary
• PAC Prediction Sets Under Covariate Shift
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Regularizing Attention Networks for Anomaly Detection in Visual Question Answering
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Uncertainty Prediction for Deep Sequential Regression Using Meta Models
• Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
• Ab-Initio Potential Energy Surfaces by Pairing GNNs with Neural Wave Functions
• No One Representation to Rule Them All: Overlapping Features of Training Methods
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Semi-Local 3D Lane Detection and Uncertainty Estimation
• A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning
• Taming the Cross Entropy Loss
• Calibration of Neural Networks using Splines
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Probabilistic Spatial Transformer Networks
• Robust Federated Learning with Attack-Adaptive Aggregation
• Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• A Targeted Universal Attack on Graph Convolutional Network
• Improving Confidence Estimates for Unfamiliar Examples
• Structured Variational Inference in Unstable Gaussian Process State Space Models
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• On Calibration of Mixup Training for Deep Neural Networks
• What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Ab initio machine learning in chemical compound space
• Provably Robust Detection of Out-of-distribution Data (almost) for free
• Predicting Adversarial Examples with High Confidence
• Impostor Networks for Fast Fine-Grained Recognition
• Progressively Complementary Network for Fisheye Image Rectification Using Appearance Flow
• Zap: Making Predictions Based on Online User Behavior
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Learnable Uncertainty under Laplace Approximations
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade
• FairCal: Fairness Calibration for Face Verification
• An Imitation Game for Learning Semantic Parsers from User Interaction
• Deep Probability Estimation
• Posterior Re-calibration for Imbalanced Datasets
• Few-shot Conformal Prediction with Auxiliary Tasks
• Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities
• Energy-Based Models with Applications to Speech and Language Processing
• Estimation of Accurate and Calibrated Uncertainties in Deterministic models
• MLOD: Awareness of Extrinsic Perturbation in Multi-LiDAR 3D Object Detection for Autonomous Driving
• False Negative Reduction in Video Instance Segmentation using Uncertainty Estimates
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Toward Metrics for Differentiating Out-of-Distribution Sets
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition
• Robust quantum dots charge autotuning using neural network uncertainty
• Variational Metric Scaling for Metric-Based Meta-Learning
• Low-Resource Multi-Granularity Academic Function Recognition Based on Multiple Prompt Knowledge
• ROAM: Random Layer Mixup for Semi-Supervised Learning in Medical Imaging
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction
• Adversarial Attack on Large Scale Graph
• A Multi-site Study of a Breast Density Deep Learning Model for Full-field Digital Mammography Images and Synthetic Mammography Images
• Unsupervised Temperature Scaling: An Unsupervised Post-Processing Calibration Method of Deep Networks
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection
• Resolving Intent Ambiguities by Retrieving Discriminative Clarifying Questions
• Star-Galaxy Separation via Gaussian Processes with Model Reduction
• Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inference
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• Intelligence plays dice: Stochasticity is essential for machine learning
• The Misclassification of Autistic Writing as AI-Generated
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
• Efficient Action Recognition Using Confidence Distillation
• MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
• Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets
• Distribution-Free Federated Learning with Conformal Predictions
• Robustness via Cross-Domain Ensembles
• Bayesian Neural Networks with Soft Evidence
• Loss Functions for Discrete Contextual Pricing with Observational Data
• Assessing Deep Neural Networks as Probability Estimators
• From Generalized zero-shot learning to long-tail with class descriptors
• Variational Depth Search in ResNets
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Uncertainty-Aware COVID-19 Detection from Imbalanced Sound Data
• Automatic Differentiation Variational Inference with Mixtures
• Structure-Aware Label Smoothing for Graph Neural Networks
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Amortized Probabilistic Detection of Communities in Graphs
• Online Safety Assurance for Deep Reinforcement Learning
• Object Tracking Using Spatio-Temporal Future Prediction
• Multi-Scale Networks for 3D Human Pose Estimation with Inference Stage Optimization
• A witness function based construction of discriminative models using Hermite polynomials
• Confident Naturalness Explanation (CNE): A Framework to Explain and Assess Patterns Forming Naturalness
• Interactive Mars Image Content-Based Search with Interpretable Machine Learning
• Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results
• Sampling Bias in Deep Active Classification: An Empirical Study
• Risk-Aware Planning by Confidence Estimation using Deep Learning-Based Perception
• Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation
• Biologically-inspired Salience Affected Artificial Neural Network (SANN)
• Towards User Guided Actionable Recourse
• Non-Parametric Calibration for Classification
• Ranking Neural Checkpoints
• Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs
• Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack
• Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• Collaborative Sampling in Generative Adversarial Networks
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Knowing what you know in brain segmentation using Bayesian deep neural networks
• Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Control
• Seeing in the dark with recurrent convolutional neural networks
• Reliable Semi-Supervised Learning when Labels are Missing at Random
• Stronger Calibration Lower Bounds via Sidestepping
• Confidence Calibration with Bounded Error Using Transformations
• High-Performance FPGA-based Accelerator for Bayesian Neural Networks
• Modeling Fine-Grained Entity Types with Box Embeddings
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
• A New Angle on L2 Regularization
• Deep Transfer Learning with Ridge Regression
• Ask-n-Learn: Active Learning via Reliable Gradient Representations for Image Classification
• Optimizing Black-box Metrics with Iterative Example Weighting
• On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
• Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialog
• Top-label calibration and multiclass-to-binary reductions
• Credit scoring using neural networks and SURE posterior probability calibration
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• Sparse MoEs meet Efficient Ensembles
• Money on the Table: Statistical information ignored by Softmax can improve classifier accuracy
• ParaNet - Using Dense Blocks for Early Inference
• Probabilistic Approach for Road-Users Detection
• Diversifying Dialog Generation via Adaptive Label Smoothing
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Doppler Spectrum Classification with CNNs via Heatmap Location Encoding and a Multi-head Output Layer
• VirAAL: Virtual Adversarial Active Learning For NLU
• Online Black-Box Confidence Estimation of Deep Neural Networks
• Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation
• Calibrated Prediction Intervals for Neural Network Regressors
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Thought Flow Nets: From Single Predictions to Trains of Model Thought
• Boldness-Recalibration for Binary Event Predictions
• Learning Augmentation Distributions using Transformed Risk Minimization
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• DeepMix: Online Auto Data Augmentation for Robust Visual Object Tracking
• The Compact Support Neural Network
• Conformance Constraint Discovery: Measuring Trust in Data-Driven Systems
• Uncertainty Propagation in Node Classification
• Performance Measurement for Deep Bayesian Neural Network
• Non-Sequential Neural Network for Simultaneous, Consistent Classification and Photometric Redshifts of OTELO Galaxies
• Deep inference of simulated strong lenses in ground-based surveys
• Calibrated Top-1 Uncertainty estimates for classification by score based models
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• Power Pooling Operators and Confidence Learning for Semi-Supervised Sound Event Detection
• Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
• Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference
• Trinary Tools for Continuously Valued Binary Classifiers
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• Generalised Zero-Shot Learning with a Classifier Ensemble over Multi-Modal Embedding Spaces
• Fast Adaptation with Linearized Neural Networks
• Image-Based Jet Analysis
• Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model
• Transfer Learning in Automated Gamma Spectral Identification
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Benchmarking CNN on 3D Anatomical Brain MRI: Architectures, Data Augmentation and Deep Ensemble Learning
• 6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference
• Machine-learned trends in mirror configurations in the Large Plasma Device
• Multivariate Deep Evidential Regression
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty
• Crop Rotation Modeling for Deep Learning-Based Parcel Classification from Satellite Time Series
• The Science of Rejection: A Research Area for Human Computation
• Structured Stochastic Gradient MCMC
• Variational Laplace for Bayesian neural networks
• Confidence Preservation Property in Knowledge Distillation Abstractions
• Generalization by Recognizing Confusion
• Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers
• The Robust Semantic Segmentation UNCV2023 Challenge Results
• Evidential Turing Processes
• Class-Similarity Based Label Smoothing for Confidence Calibration
• UNGOML: Automated Classification of unsafe Usages in Go
• Consistent Accelerated Inference via Confident Adaptive Transformers
• ModelPred: A Framework for Predicting Trained Model from Training Data
• MIMIR: Deep Regression for Automated Analysis of UK Biobank Body MRI
• Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain
• Understanding Classifier Mistakes with Generative Models
• Revealing the Distributional Vulnerability of Discriminators by Implicit Generators
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Self Training with Ensemble of Teacher Models
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
• On Class Imbalance and Background Filtering in Visual Relationship Detection
• Teaching Uncertainty Quantification in Machine Learning through Use Cases
• The Case for Claim Difficulty Assessment in Automatic Fact Checking
• MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator
• Bayesian deep learning of affordances from RGB images
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Pathologies in priors and inference for Bayesian transformers
• Natural Attribute-based Shift Detection
• On the Calibration of Nested Dichotomies for Large Multiclass Tasks
• Know Where To Drop Your Weights: Towards Faster Uncertainty Estimation
• Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
• Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration
• Assessing The Importance Of Colours For CNNs In Object Recognition
• Long Short-Term Sample Distillation
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• Multi-Sample Online Learning for Spiking Neural Networks based on Generalized Expectation Maximization
• An analysis of gamma-ray data collected at traffic intersections in Northern Virginia
• White Paper Assistance: A Step Forward Beyond the Shortcut Learning
• Learning to Predict Error for MRI Reconstruction
• MisMatch: Calibrated Segmentation via Consistency on Differential Morphological Feature Perturbations with Limited Labels
• MOMO -- Deep Learning-driven classification of external DICOM studies for PACS archivation
• Anomaly-Aware Semantic Segmentation by Leveraging Synthetic-Unknown Data
• A Grounded Well-being Conversational Agent with Multiple Interaction Modes: Preliminary Results
• Uncertainty estimation under model misspecification in neural network regression
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms
• Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs against Query-Based Attacks
• Learning from Matured Dumb Teacher for Fine Generalization
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model
• Evaluating Adversarial Attacks on ImageNet: A Reality Check on Misclassification Classes