6 papers
CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
Tianneng Shi, Robin Rheem, Dongwei Jiang +13
AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity e…
Supplement Generation Training for Enhancing Agentic Task Performance
Young Min Cho, Daniele Bonadiman, Divya Bhargavi +8
Training large foundation models for agentic tasks is increasingly impractical due to the high computational costs, long iteration cycles, and rapid obsolescence as new models are…
Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
Dongwei Jiang, Alvin Zhang, Andrew Wang +2
Recent studies have shown LLMs possess some ability to improve their responses when given external feedback. However, it remains unclear how effectively and thoroughly these models…
RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoning
Dongwei Jiang, Guoxuan Wang, Yining Lu +5
The reasoning steps generated by LLMs might be incomplete, as they mimic logical leaps common in everyday communication found in their pre-training data: underlying rationales are…
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez +7
Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thin…
Benchmarking Language Model Creativity: A Case Study on Code Generation
Yining Lu, Dixuan Wang, Tianjian Li +4
As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key character…