Quo Vadis ChatGPT? From Large Language Models to Large Knowledge Models

Abstract

The startling success of ChatGPT and other large language models (LLMs) using transformer-based generative neural network architecture in applications such as natural language processing and image synthesis has many researchers excited about potential opportunities in process systems engineering (PSE). The almost human-like performance of LLMs in these areas is indeed very impressive, surprising, and a major breakthrough. Their capabilities are very useful in certain tasks, such as writing first drafts of documents, code writing assistance, text summarization, etc. However, their success is limited in highly scientific domains as they cannot yet reason, plan, or explain due to their lack of in-depth domain knowledge. This is a problem in domains such as chemical engineering as they are governed by fundamental laws of physics and chemistry (and biology), constitutive relations, and highly technical knowledge about materials, processes, and systems. Although purely data-driven machine learning has its immediate uses, the long-term success of AI in scientific and engineering domains would depend on developing hybrid AI systems that use first principles and technical knowledge effectively. We call these hybrid AI systems Large Knowledge Models (LKMs), as they will not be limited to only NLP-based techniques or NLP-like applications. In this paper, we discuss the challenges and opportunities in developing such systems in chemical engineering.

References in corpus (21)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Gemini: A Family of Highly Capable Multimodal Models
• LaMDA: Language Models for Dialog Applications
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• The ALAMO approach to machine learning
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• ChemCrow: Augmenting large-language models with chemistry tools
• Textbooks Are All You Need
• Larger language models do in-context learning differently
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• Scientific Large Language Models: A Survey on Biological & Chemical Domains
• Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc
• What's in an embedding? Would a rose by any embedding smell as sweet?

Cited by in corpus (1)

• Stable diffusion models reveal a persisting human and AI gap in visual creativity