HPC-GPT: Integrating Large Language Model for High-Performance Computing

Abstract

Large Language Models (LLMs), including the LLaMA model, have exhibited their efficacy across various general-domain natural language processing (NLP) tasks. However, their performance in high-performance computing (HPC) domain tasks has been less than optimal due to the specialized expertise required to interpret the model responses. In response to this challenge, we propose HPC-GPT, a novel LLaMA-based model that has been supervised fine-tuning using generated QA (Question-Answer) instances for the HPC domain. To evaluate its effectiveness, we concentrate on two HPC tasks: managing AI models and datasets for HPC, and data race detection. By employing HPC-GPT, we demonstrate comparable performance with existing methods on both tasks, exemplifying its excellence in HPC-related scenarios. Our experiments on open-source benchmarks yield extensive results, underscoring HPC-GPT's potential to bridge the performance gap between LLMs and HPC-specific tasks. With HPC-GPT, we aim to pave the way for LLMs to excel in HPC domains, simplifying the utilization of language models in complex computing applications.

arXiv comment9 pages

References in corpus (21)

• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoRA: Low-Rank Adaptation of Large Language Models
• PaLM: Scaling Language Modeling with Pathways
• Cross-lingual Language Model Pretraining
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Evaluating Large Language Models Trained on Code
• CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Mixed supervision for surface-defect detection: from weakly to fully supervised learning
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Generative Data Augmentation for Commonsense Reasoning
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
• LM4HPC: Towards Effective Language Model Application in High-Performance Computing
• LLOV: A Fast Static Data-Race Checker for OpenMP Programs
• Data Race Detection Using Large Language Models
• Learning to Parallelize with OpenMP by Augmented Heterogeneous AST Representation
• DataRaceBench V1.4.1 and DataRaceBench-ML V0.1: Benchmark Suites for Data Race Detection

Cited by in corpus (5)

• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• OMPGPT: A Generative Pre-trained Transformer Model for OpenMP
• Large Language Model Integrated Healthcare Cyber-Physical Systems Architecture
• LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks
• LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation