Optimising the Input Image to Improve Visual Relationship Detection

Abstract

Visual Relationship Detection is defined as, given an image composed of a subject and an object, the correct relation is predicted. To improve the visual part of this difficult problem, ten preprocessing methods were tested to determine whether the widely used Union method yields the optimal results. Therefore, focusing solely on predicate prediction, no object detection and linguistic knowledge were used to prevent them from affecting the comparison results. Once fine-tuned, the Visual Geometry Group models were evaluated using Recall@1, per-predicate recall, activation maximisations, class activation maps, and error analysis. From this research it was found that using preprocessing methods such as the Union-Without-Background-and-with-Binary-mask (Union-WB-and-B) method yields significantly better results than the widely used Union method since, as designed, it enables the Convolutional Neural Network to also identify the subject and object in the convolutional layers instead of solely in the fully-connected layers.

References in corpus (6)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Understanding Neural Networks Through Deep Visualization
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Object Detectors Emerge in Deep Scene CNNs
• Recent Advances in Convolutional Neural Networks
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks