Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Abstract

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

arXiv commentAccepted for publication in Nature Communications

References in corpus (10)

• Methods for Interpreting and Understanding Deep Neural Networks
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Machine Learning of Accurate Energy-Conserving Molecular Force Fields
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• How to Explain Individual Classification Decisions
• Towards Exact Molecular Dynamics Simulations with Machine-Learned Force Fields
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Explaining the Unique Nature of Individual Gait Patterns with Deep Learning

Cited by in corpus (87)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Shortcut Learning in Deep Neural Networks
• Machine Learning Force Fields
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Explainable Machine Learning for Scientific Insights and Discoveries
• Machine learning for molecular simulation
• A Unifying Review of Deep and Shallow Anomaly Detection
• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
• Towards Explainable Artificial Intelligence
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• Artificial Intelligence and Big Data in Entrepreneurship: A New Era Has Begun
• SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
• Explanations in Autonomous Driving: A Survey
• The Debate Over Understanding in AI's Large Language Models
• Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability
• A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts
• Higher-Order Explanations of Graph Neural Networks via Relevant Walks
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• The Pragmatic Turn in Explainable Artificial Intelligence (XAI)
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• Neural Network Potentials for Chemistry: Concepts, Applications and Prospects
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Toward Explainable AI for Regression Models
• From Clustering to Cluster Explanations via Neural Networks
• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• Towards a Collective Agenda on AI for Earth Science Data Analysis
• Implementing local-explainability in Gradient Boosting Trees: Feature Contribution
• On the Explanation of Machine Learning Predictions in Clinical Gait Analysis
• Explaining and Interpreting LSTMs
• Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• How we do things with words: Analyzing text as social and cultural data
• Building and Interpreting Deep Similarity Models
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
• Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria
• Machine learning astrophysics from 21 cm lightcones: impact of network architectures and signal contamination
• Machine learning methods for prediction of cancer driver genes: a survey paper
• Survey of XAI in digital pathology
• An XAI framework for robust and transparent data-driven wind turbine power curve models
• Comprehensible Artificial Intelligence on Knowledge Graphs: A survey
• Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
• Machine Learning for Observational Cosmology
• The Clever Hans Effect in Unsupervised Learning
• A Survey on Neural Network Interpretability
• White Box Methods for Explanations of Convolutional Neural Networks in Image Classification Tasks
• Discrete and continuous representations and processing in deep learning: Looking forward
• Vulnerabilities of Connectionist AI Applications: Evaluation and Defence
• Machine Learning Workflow to Explain Black-box Models for Early Alzheimer's Disease Classification Evaluated for Multiple Datasets
• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
• Explainable multiple abnormality classification of chest CT volumes
• Insights Into the Inner Workings of Transformer Models for Protein Function Prediction
• SLISEMAP: Supervised dimensionality reduction through local explanations
• DeepView: Visualizing Classification Boundaries of Deep Neural Networks as Scatter Plots Using Discriminative Dimensionality Reduction
• Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
• OnRAMP for Regulating AI in Medical Products
• Machine Learning in Biomechanics: Key Applications and Limitations in Walking, Running, and Sports Movements
• Expressive Explanations of DNNs by Combining Concept Analysis with ILP
• Construction of Machine Learned Force Fields with Quantum Chemical Accuracy: Applications and Chemical Insights
• Next2You: Robust Copresence Detection Based on Channel State Information
• PredDiff: Explanations and Interactions from Conditional Expectations
• The Importance of Being Interpretable: Toward An Understandable Machine Learning Encoder for Galaxy Cluster Cosmology
• MAPS-X: Explainable Multi-Robot Motion Planning via Segmentation
• Reliability Scores from Saliency Map Clusters for Improved Image-based Harvest-Readiness Prediction in Cauliflower
• Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI
• i-Align: an interpretable knowledge graph alignment model
• Human-Centered Evaluation of XAI Methods
• A Conceptual Framework for Establishing Trust in Real World Intelligent Systems
• Study on the Helpfulness of Explainable Artificial Intelligence
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Computationally efficient neural network classifiers for next generation closed loop neuromodulation therapy -- a case study in epilepsy
• On The Coherence of Quantitative Evaluation of Visual Explanations
• SE3D: A Framework For Saliency Method Evaluation In 3D Imaging
• What Do Large Language Models Know? Tacit Knowledge as a Potential Causal-Explanatory Structure
• Exploring complex pattern formation with convolutional neural networks
• Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces
• Background Bias Assessment and Mitigation in Plant Disease Detection Using EfficientNet-B0 with Image Processing
• Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems
• Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation
• P-TAME: Explain Any Image Classifier with Trained Perturbations
• Towards Desiderata-Driven Design of Visual Counterfactual Explainers
• How can we trust opaque systems? Criteria for robust explanations in XAI