Identifying Object States in Cooking-Related Images

Abstract

Understanding object states is as important as object recognition for robotic task planning and manipulation. To our knowledge, this paper explicitly introduces and addresses the state identification problem in cooking related images for the first time. In this paper, objects and ingredients in cooking videos are explored and the most frequent objects are analyzed. Eleven states from the most frequent cooking objects are examined and a dataset of images containing those objects and their states is created. As a solution to the state identification problem, a Resnet based deep model is proposed. The model is initialized with Imagenet weights and trained on the dataset of eleven classes. The trained state identification model is evaluated on a subset of the Imagenet dataset and state labels are provided using a combination of the model with manual checking. Moreover, an individual model is fine-tuned for each object in the dataset using the weights from the initially trained model and object-specific images, where significant improvement is demonstrated.

arXiv comment7 pages, 8 figures

References in corpus (1)

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by in corpus (14)

• A Survey of Knowledge Representation in Service Robotics
• Cooking State Recognition from Images Using Inception Architecture
• Rethinking Cooking State Recognition with Vision Transformers
• Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization
• Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation through Grounded Anomaly Classification and Recovery Policies
• Tuned Inception V3 for Recognizing States of Cooking Ingredients
• State Classification of Cooking Objects Using a VGG CNN
• Task Planning with a Weighted Functional Object-Oriented Network
• Joint Object and State Recognition using Language Knowledge
• OnionBot: A System for Collaborative Computational Cooking
• VGG Fine-tuning for Cooking State Recognition
• Synthetic Image Augmentation for Improved Classification using Generative Adversarial Networks
• Robot Gaining Accurate Pouring Skills through Self-Supervised Learning and Generalization
• Classifying States of Cooking Objects Using Convolutional Neural Network