Do Convolutional Neural Networks Learn Class Hierarchy?

Abstract

Convolutional Neural Networks (CNNs) currently achieve state-of-the-art accuracy in image classification. With a growing number of classes, the accuracy usually drops as the possibilities of confusion increase. Interestingly, the class confusion patterns follow a hierarchical structure over the classes. We present visual-analytics methods to reveal and analyze this hierarchy of similar classes in relation with CNN-internal data. We found that this hierarchy not only dictates the confusion patterns between the classes, it furthermore dictates the learning behavior of CNNs. In particular, the early layers in these networks develop feature detectors that can separate high-level groups of classes quite well, even after a few training epochs. In contrast, the latter layers require substantially more epochs to develop specialized feature detectors that can separate individual classes. We demonstrate how these insights are key to significant improvement in accuracy by designing hierarchy-aware CNNs that accelerate model convergence and alleviate overfitting. We further demonstrate how our methods help in identifying various quality issues in the training data.

Video demo at https://vimeo.com/228263798

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Striving for Simplicity: The All Convolutional Net
• Understanding Neural Networks Through Deep Visualization
• Object Detectors Emerge in Deep Scene CNNs
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Direct-Manipulation Visualization of Deep Networks
• Understanding Intra-Class Knowledge Inside CNN
• A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
• Decision Forests, Convolutional Networks and the Models in-Between

Cited by in corpus (50)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation
• The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
• ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion
• Diagnosing Concept Drift with Visual Analytics
• Survey of XAI in digital pathology
• A User-based Visual Analytics Workflow for Exploratory Model Analysis
• Diagnosing Ensemble Few-Shot Classifiers
• NNVA: Neural Network Assisted Visual Analysis of Yeast Cell Polarization Simulation
• Classifier-Guided Visual Correction of Noisy Labels for Image Classification Tasks
• DeepTracker: Visualizing the Training Process of Convolutional Neural Networks
• Machine learning approaches for automatic defect detection in photovoltaic systems
• InstanceFlow: Visualizing the Evolution of Classifier Confusion on the Instance Level
• iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models
• Self-Damaging Contrastive Learning
• A Survey of Visual Analytics Techniques for Machine Learning
• Look-into-Object: Self-supervised Structure Modeling for Object Recognition
• Melody: Generating and Visualizing Machine Learning Model Summary to Understand Data and Classifiers Together
• DECE: Decision Explorer with Counterfactual Explanations for Machine Learning Models
• CNNPruner: Pruning Convolutional Neural Networks with Visual Analytics
• VATLD: A Visual Analytics System to Assess, Understand and Improve Traffic Light Detection
• Not just a matter of semantics: the relationship between visual similarity and semantic similarity
• Classifying the classifier: dissecting the weight space of neural networks
• RuleMatrix: Visualizing and Understanding Classifiers with Rules
• M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
• E-PixelHop: An Enhanced PixelHop Method for Object Classification
• Learning with Hierarchical Complement Objective
• Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs
• Prediction Scores as a Window into Classifier Behavior
• Coarse-to-Fine Curriculum Learning
• VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level
• A Visual Analytics Framework for Explaining and Diagnosing Transfer Learning Processes
• Hierarchical nucleation in deep neural networks
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• LAMVI-2: A Visual Tool for Comparing and Tuning Word Embedding Models
• Dependency Decomposition and a Reject Option for Explainable Models
• United We Learn Better: Harvesting Learning Improvements From Class Hierarchies Across Tasks
• Bottom-up Hierarchical Classification Using Confusion-based Logit Compression
• CACTUS: Detecting and Resolving Conflicts in Objective Functions
• Exploiting Class Similarity for Machine Learning with Confidence Labels and Projective Loss Functions
• Nested Learning For Multi-Granular Tasks
• Visualizing Classification Structure of Large-Scale Classifiers
• Decision Propagation Networks for Image Classification
• Visual Summary of Value-level Feature Attribution in Prediction Classes with Recurrent Neural Networks
• Visual Reasoning of Feature Attribution with Deep Recurrent Neural Networks
• A class-driven hierarchical ResNet for classification of multispectral remote sensing images