Towards Recognizing New Semantic Concepts in New Visual Domains

Abstract

Deep learning models heavily rely on large scale annotated datasets for training. Unfortunately, datasets cannot capture the infinite variability of the real world, thus neural networks are inherently limited by the restricted visual and semantic information contained in their training set. In this thesis, we argue that it is crucial to design deep architectures that can operate in previously unseen visual domains and recognize novel semantic concepts. In the first part of the thesis, we describe different solutions to enable deep models to generalize to new visual domains, by transferring knowledge from a labeled source domain(s) to a domain (target) where no labeled data are available. We will show how variants of batch-normalization (BN) can be applied to different scenarios, from domain adaptation when source and target are mixtures of multiple latent domains, to domain generalization, continuous domain adaptation, and predictive domain adaptation, where information about the target domain is available only in the form of metadata. In the second part of the thesis, we show how to extend the knowledge of a pretrained deep model to new semantic concepts, without access to the original training set. We address the scenarios of sequential multi-task learning, using transformed task-specific binary masks, open-world recognition, with end-to-end training and enforced clustering, and incremental class learning in semantic segmentation, where we highlight and address the problem of the semantic shift of the background class. In the final part, we tackle a more challenging problem: given images of multiple domains and semantic categories (with their attributes), how to build a model that recognizes images of unseen concepts in unseen domains? We also propose an approach based on domain and semantic mixing of inputs and features, which is a first, promising step towards solving this problem.

Ph.D. thesis. Sapienza University of Rome (2020)

References in corpus (41)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Continuous control with deep reinforcement learning
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning Transferable Features with Deep Adaptation Networks
• Deep Visual Domain Adaptation: A Survey
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Theoretical Models of Learning to Learn
• Recent Trends in Deep Learning Based Natural Language Processing
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Fine-Grained Visual Classification of Aircraft
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Domain Separation Networks
• Domain Generalization via Invariant Feature Representation
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Asymmetric Tri-training for Unsupervised Domain Adaptation
• Revisiting Batch Normalization For Practical Domain Adaptation
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Measuring Catastrophic Forgetting in Neural Networks
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• ImageNet pre-trained models with batch normalization
• Continual Unsupervised Representation Learning
• Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Budget-Aware Adapters for Multi-Domain Learning
• Inferring Latent Domains for Unsupervised Deep Domain Adaptation
• Learning Deep NBNN Representations for Robust Place Categorization
• Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
• Robot In a Room: Toward Perfect Object Recognition in Closed Environments
• Online Open World Recognition
• Boosting Binary Masks for Multi-Domain Learning through Affine Transformations
• Rethinking Continual Learning for Autonomous Agents and Robots