VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities
arXiv:2412.18161 · doi:10.1088/2632-2153/add9e4
Abstract
Scientific user facilities, such as synchrotron beamlines, are equipped with a wide array of hardware and software tools that require a codebase for human-computer-interaction. This often necessitates developers to be involved to establish connection between users/researchers and the complex instrumentation. The advent of generative AI presents an opportunity to bridge this knowledge gap, enabling seamless communication and efficient experimental workflows. Here we present a modular architecture for the Virtual Scientific Companion (VISION) by assembling multiple AI-enabled cognitive blocks that each scaffolds large language models (LLMs) for a specialized task. With VISION, we performed LLM-based operation on the beamline workstation with low latency and demonstrated the first voice-controlled experiment at an X-ray scattering beamline. The modular and scalable architecture allows for easy adaptation to new instrument and capabilities. Development on natural language-based scientific experimentation is a building block for an impending future where a science exocortex -- a synthetic extension to the cognition of scientists -- may radically transform scientific practice and discovery.
References in corpus (26)
- Training language models to follow instructions with human feedback
- LoRA: Low-Rank Adaptation of Large Language Models
- Robust Speech Recognition via Large-Scale Weak Supervision
- ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Accelerating computational materials discovery with artificial intelligence and cloud high-performance computing: from large-scale screening to experimental validation
- PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
- Qwen2.5-Coder Technical Report
- Language agents achieve superhuman synthesis of scientific knowledge
- Agent Laboratory: Using LLM Agents as Research Assistants
- Domain-specific ChatBots for Science using Embeddings
- Hypothesis Generation with Large Language Models
- LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation
- LLMatDesign: Autonomous Materials Discovery with Large Language Models
- DiffractGPT: Atomic Structure Determination from X-ray Diffraction Patterns using Generative Pre-trained Transformer
- Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders
- LitLLM: A Toolkit for Scientific Literature Review
- Aviary: training language agents on challenging scientific tasks
- AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
- Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systems
- Large Language Model-Guided Prediction Toward Quantum Materials Synthesis
- Specifications: The missing link to making the development of LLM systems an engineering discipline
- Virtual Scientific Companion for Synchrotron Beamlines: A Prototype
- Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback