HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Abstract

Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously. Considering large language models (LLMs) have exhibited exceptional abilities in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks, with language serving as a generic interface to empower this. Based on this philosophy, we present HuggingGPT, an LLM-powered agent that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards the realization of artificial general intelligence.

Cited by in corpus (27)

• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
• Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
• A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
• Woodpecker: Hallucination Correction for Multimodal Large Language Models
• Evaluation of large language models for discovery of gene set function
• General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• A Large Language Model Approach to Educational Survey Feedback Analysis
• Kartezio: Evolutionary Design of Explainable Pipelines for Biomedical Image Analysis
• C2Ideas: Supporting Creative Interior Color Design Ideation with Large Language Model
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)
• The Impact of Modern AI in Metadata Management
• Domain-specific ChatBots for Science using Embeddings
• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• Large Knowledge Model: Perspectives and Challenges
• PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
• Teola: Towards End-to-End Optimization of LLM-based Applications
• Complex QA and language models hybrid architectures, Survey
• Flows: Building Blocks of Reasoning and Collaborating AI
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
• QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Models
• Tool-Augmented LLMs as a Universal Interface for IDEs
• Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelines
• ITCMA: A Generative Agent Based on a Computational Consciousness Structure