Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT

Abstract

Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages. To address these challenges, we investigate the efficiency of Large Language Models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed three user studies in two scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.

References in corpus (21)

• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Evaluating Large Language Models Trained on Code
• Showing Academic Performance Predictions during Term Planning: Effects on Students' Decisions, Behaviors, and Preferences
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• LaMDA: Language Models for Dialog Applications
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Code Llama: Open Foundation Models for Code
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• The Placebo Effect of Artificial Intelligence in Human-Computer Interaction
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• Teaching Large Language Models to Self-Debug
• A Community Roadmap for Scientific Workflows Research and Development
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
• Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
• Human-in-the-Loop Schema Induction
• Human-in-the-Loop through Chain-of-Thought
• ChatGPT as your Personal Data Scientist

Cited by in corpus (1)

• Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT