FoodSAM: Any Food Segmentation

Abstract

In this paper, we explore the zero-shot capability of the Segment Anything Model (SAM) for food image segmentation. To address the lack of class-specific information in SAM-generated masks, we propose a novel framework, called FoodSAM. This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic segmentation quality. Besides, we recognize that the ingredients in food can be supposed as independent individuals, which motivated us to perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Drawing inspiration from the recent success of promptable segmentation, we also extend FoodSAM to promptable segmentation, supporting various prompt variants. Consequently, FoodSAM emerges as an all-encompassing solution capable of segmenting food items at multiple levels of granularity. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. Extensive experiments demonstrate the feasibility and impressing performance of FoodSAM, validating SAM's potential as a prominent and influential tool within the domain of food image segmentation. We release our code at https://github.com/jamesjg/FoodSAM.

arXiv commentCode is available at https://github.com/jamesjg/FoodSAM

References in corpus (20)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• On the Opportunities and Risks of Foundation Models
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Pre-trained Models for Natural Language Processing: A Survey
• Emergent Abilities of Large Language Models
• Segment Anything
• OCNet: Object Context Network for Scene Parsing
• A Survey on Instance Segmentation: State of the art
• Florence: A New Foundation Model for Computer Vision
• Segment Everything Everywhere All at Once
• Pretrained Transformers as Universal Computation Engines
• Applications of knowledge graphs for food science and industry
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Fast Segment Anything
• Recognize Anything: A Strong Image Tagging Model
• Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions

Cited by in corpus (3)

• Navigating Weight Prediction with Diet Diary
• Recognizing Multiple Ingredients in Food Images Using a Single-Ingredient Classification Model
• A SAM based Tool for Semi-Automatic Food Annotation