102 citations · 366 across the 31 of their papers we have counts for
31 papers
AL-Bench: A Benchmark for Automatic Logging
Boyin Tan, Junjielong Xu, Zhouruixing Zhu +1
Logging, the practice of inserting log statements into source code, is critical for improving software reliability. Recently, language model-based techniques have been developed to…
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12
Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In…
SPES: Towards Optimizing Performance-Resource Trade-Off for Serverless Functions
Cheryl Lee, Zhouruixing Zhu, Tianyi Yang +4
As an emerging cloud computing deployment paradigm, serverless computing is gaining traction due to its efficiency and ability to harness on-demand cloud resources. However, a sign…
Go Static: Contextualized Logging Statement Generation
Yichen Li, Yintong Huo, Renyi Zhong +6
Logging practices have been extensively investigated to assist developers in writing appropriate logging statements for documenting software behaviors. Although numerous automatic…
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
Yuxuan Wan, Wenxuan Wang, Yiliu Yang +5
We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4. Despite LLMs' p…
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Zhihan Jiang, Jinyang Liu, Zhuangbin Chen +6
Log parsing transforms log messages into structured formats, serving as the prerequisite step for various log analysis tasks. Although a variety of log parsing approaches have been…
An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software
Wenxuan Wang, Jingyuan Huang, Jen-tse Huang +4
The exponential growth of social media platforms has brought about a revolution in communication and content dissemination in human society. Nevertheless, these platforms are being…
AutoLog: A Log Sequence Synthesis Framework for Anomaly Detection
Yintong Huo, Yichen Li, Yuxin Su +3
The rapid progress of modern computing systems has led to a growing interest in informative run-time logs. Various log-based anomaly detection techniques have been proposed to ensu…
Automated Testing and Improvement of Named Entity Recognition Systems
Boxi Yu, Yiyan Hu, Qiuyang Mang +2
Named entity recognition (NER) systems have seen rapid progress in recent years due to the development of deep neural networks. These systems are widely used in various natural lan…
Hue: A User-Adaptive Parser for Hybrid Logs
Junjielong Xu, Qiuai Fu, Zhouruixing Zhu +4
Log parsing, which extracts log templates from semi-structured logs and produces structured logs, is the first and the most critical step in automated log analysis. While existing…
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang +4
Safety lies at the core of the development of Large Language Models (LLMs). There is ample work on aligning LLMs with human ethics and preferences, including data filtering in pret…
ROME: Testing Image Captioning Systems via Recursive Object Melting
Boxi Yu, Zhiqing Zhong, Jiaqi Li +3
Image captioning (IC) systems aim to generate a text description of the salient objects in an image. In recent years, IC systems have been increasingly integrated into our daily li…
Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study
Yichen Li, Yintong Huo, Zhihan Jiang +5
Automated logging statement generation supports developers in documenting critical software runtime behavior. Given the great success in natural language generation and programming…
Validating Multimedia Content Moderation Software via Semantic Fusion
Wenxuan Wang, Jingyuan Huang, Chang Chen +5
The exponential growth of social media platforms, such as Facebook and TikTok, has revolutionized communication and content publication in human society. Users on these platforms c…
BiasAsker: Measuring the Bias in Conversational AI System
Yuxuan Wan, Wenxuan Wang, Pinjia He +3
Powered by advanced Artificial Intelligence (AI) techniques, conversational AI systems, such as ChatGPT and digital assistants like Siri, have been widely deployed in daily life. H…
Incident-aware Duplicate Ticket Aggregation for Cloud Systems
Jinyang Liu, Shilin He, Zhuangbin Chen +10
In cloud systems, incidents are potential threats to customer satisfaction and business revenue. When customers are affected by incidents, they often request customer support servi…
MTTM: Metamorphic Testing for Textual Content Moderation Software
Wenxuan Wang, Jen-tse Huang, Weibin Wu +5
The exponential growth of social media platforms such as Twitter and Facebook has revolutionized textual communication and textual content publication in human society. However, th…
Automated Testing of Image Captioning Systems
Boxi Yu, Zhiqing Zhong, Xinran Qin +3
Image captioning (IC) systems, which automatically generate a text description of the salient objects in an image (real or synthetic), have seen great progress over the past few ye…
AEON: A Method for Automatic Evaluation of NLP Test Cases
Jen-tse Huang, Jianping Zhang, Wenxuan Wang +3
Due to the labor-intensive nature of manual test oracle construction, various automated testing techniques have been proposed to enhance the reliability of Natural Language Process…
Empirical Standards for Software Engineering Research
Paul Ralph, Nauman bin Ali, Sebastian Baltes +39
Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer…
A Survey on Automated Log Analysis for Reliability Engineering
Shilin He, Pinjia He, Zhuangbin Chen +3
Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanis…
Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics
Jieming Zhu, Shilin He, Pinjia He +2
Logs have been widely adopted in software system development and maintenance because of the rich runtime information they record. In recent years, the increase of software size and…
Testing Machine Translation via Referential Transparency
Pinjia He, Clara Meister, Zhendong Su
Machine translation software has seen rapid progress in recent years due to the advancement of deep neural networks. People routinely use machine translation software in their dail…
Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression
Jinyang Liu, Jieming Zhu, Shilin He +3
System logs record detailed runtime information of software systems and are used as the main data source for many tasks around software engineering. As modern software systems are…