7 papers · 1 filter
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12
Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In…
AL-Bench: A Benchmark for Automatic Logging
Boyin Tan, Junjielong Xu, Zhouruixing Zhu +1
Logging, the practice of inserting log statements into source code, is critical for improving software reliability. Recently, language model-based techniques have been developed to…
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
Yuxuan Wan, Wenxuan Wang, Yiliu Yang +5
We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4. Despite LLMs' p…
SPES: Towards Optimizing Performance-Resource Trade-Off for Serverless Functions
Cheryl Lee, Zhouruixing Zhu, Tianyi Yang +4
As an emerging cloud computing deployment paradigm, serverless computing is gaining traction due to its efficiency and ability to harness on-demand cloud resources. However, a sign…
Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study
Yichen Li, Yintong Huo, Zhihan Jiang +5
Automated logging statement generation supports developers in documenting critical software runtime behavior. Given the great success in natural language generation and programming…
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Zhihan Jiang, Jinyang Liu, Zhuangbin Chen +6
Log parsing transforms log messages into structured formats, serving as the prerequisite step for various log analysis tasks. Although a variety of log parsing approaches have been…