Semantic Image Cropping

Abstract

Automatic image cropping techniques are commonly used to enhance the aesthetic quality of an image; they do it by detecting the most beautiful or the most salient parts of the image and removing the unwanted content to have a smaller image that is more visually pleasing. In this thesis, I introduce an additional dimension to the problem of cropping, semantics. I argue that image cropping can also enhance the image's relevancy for a given entity by using the semantic information contained in the image. I call this problem, Semantic Image Cropping. To support my argument, I provide a new dataset containing 100 images with at least two different entities per image and four ground truth croppings collected using Amazon Mechanical Turk. I use this dataset to show that state-of-the-art cropping algorithms that only take into account aesthetics do not perform well in the problem of semantic image cropping. Additionally, I provide a new deep learning system that takes not just aesthetics but also semantics into account to generate image croppings, and I evaluate its performance using my new semantic cropping dataset, showing that using the semantic information of an image can help to produce better croppings.

References in corpus (7)

• Deep Residual Learning for Image Recognition
• Focal Loss for Dense Object Detection
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Feature Pyramid Networks for Object Detection
• Visualizing and Understanding Convolutional Networks
• Deep Cropping via Attention Box Prediction and Aesthetics Assessment
• Quantitative Analysis of Automatic Image Cropping Algorithms: A Dataset and Comparative Study