Preferences Prediction using a Gallery of Mobile Device based on Scene Recognition and Object Detection

Abstract

In this paper user modeling task is examined by processing a gallery of photos and videos on a mobile device. We propose novel engine for user preference prediction based on scene recognition, object detection and facial analysis. At first, all faces in a gallery are clustered and all private photos and videos with faces from large clusters are processed on the embedded system in offline mode. Other photos may be sent to the remote server to be analyzed by very deep models. The visual features of each photo are obtained from scene recognition and object detection models. These features are aggregated into a single user descriptor in the neural attention block. The proposed pipeline is implemented for the Android mobile platform. Experimental results with a subset of Photo Event Collection, Web Image Dataset for Event Recognition and Amazon Fashion datasets demonstrate the possibility to process images very efficiently without significant accuracy degradation. The source code of Android mobile application is publicly available at https://github.com/HSE-asavchenko/mobile-visual-preferences.

arXiv comment19 pages; 9 figures, preprint submitter to Pattern Recognition journal

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Learnable pooling with Context Gating for video classification
• Efficient Facial Representations for Age, Gender and Identity Recognition in Organizing Photo Albums using Multi-output CNN
• Deep Learning based Large Scale Visual Recommendation and Search for E-Commerce
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Maximum A Posteriori Estimation of Distances Between Deep Features in Still-to-Video Face Recognition
• Maximum Likelihood Directed Enumeration Method in Piecewise-Regular Object Recognition
• Beyond Classification: Latent User Interests Profiling from Visual Contents Analysis

Cited by in corpus (1)

• Event Recognition with Automatic Album Detection based on Sequential Processing, Neural Attention and Image Captioning