Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index

Abstract

This study introduces a new methodology for an Inference Index (InI), called INFerence INdex In Testing model Effectiveness methodology (INFINITE), aiming to evaluate the performance of Large Language Models (LLMs) in code generation tasks. The InI index provides a comprehensive assessment focusing on three key components: efficiency, consistency, and accuracy. This approach encapsulates time-based efficiency, response quality, and the stability of model outputs, offering a thorough understanding of LLM performance beyond traditional accuracy metrics. We applied this methodology to compare OpenAI's GPT-4o (GPT), OpenAI-o1 pro (OAI1), and OpenAI-o3 mini-high (OAI3) in generating Python code for the Long-Short-Term-Memory (LSTM) model to forecast meteorological variables such as temperature, relative humidity and wind velocity. Our findings demonstrate that GPT outperforms OAI1 and performs comparably to OAI3 regarding accuracy and workflow efficiency. The study reveals that LLM-assisted code generation can produce results similar to expert-designed models with effective prompting and refinement. GPT's performance advantage highlights the benefits of widespread use and user feedback.

20 pages, 6 figures

References in corpus (7)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Large Language Models are Zero-Shot Reasoners
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Quantifying Relevance in Learning and Inference