FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries

Abstract

Intelligence is key to advancing integrated circuit (IC) fabrication. Recent breakthroughs in Large Multimodal Models (LMMs) have unlocked extraditionary abilities in understanding images and text, fostering intelligent fabrication. Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication large multimodal model for wafer defect knowledge query. FabGPT manifests expertise in conducting defect detection in Scanning Electron Microscope (SEM) images, performing root cause analysis, and providing expert Q&A on fabrication processes. FabGPT matches enhanced multimodal features to automatically detect minute defects under complex wafer backgrounds and reduce the subjectivity of manual threshold settings. Besides, the proposed modulation module and interactive corpus training strategy embed wafer defect knowledge into the pre-trained model, effectively balancing Q&A queries related to defect knowledge and original knowledge and mitigating the modality bias issues. Experiments on in-house fab data show that FabGPT achieves significant performance improvement in wafer defect detection and knowledge querying.

Published in ACM/IEEE International Conference On Computer Aided Design (ICCAD) 2024. Corresponding Author: Qi Sun ([email protected])

References in corpus (14)

• Learning Transferable Visual Models From Natural Language Supervision
• LLaMA: Open and Efficient Foundation Language Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System
• Bidirectional Encoder Representations from Transformers (BERT): A sentiment analysis odyssey
• LISA: Reasoning Segmentation via Large Language Model
• Explainable Deep Few-shot Anomaly Detection with Deviation Networks
• Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection
• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
• Prototypical Residual Networks for Anomaly Detection and Localization
• Automated Semiconductor Defect Inspection in Scanning Electron Microscope Images: a Systematic Review
• Explicit Boundary Guided Semi-Push-Pull Contrastive Learning for Supervised Anomaly Detection
• Symbol tuning improves in-context learning in language models