29 papers
JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making
Zhongwei Xie, Xingsu Li, Jingqiao Yu +8
Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark…
Jev-ToM: Evaluating Theory of Mind in Typed Decision Models
Kawai Chung, Chunkit Chan, Yuxuan Liu +9
Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language…
Jev-Guard: Fast Routing and Auditing Agent Actions over Editable Policy Trees
Wenbin Hu, Huihao Jing, Haochen Shi +9
Agent harnesses require a low-latency guardrail model together with customizable audit policies. However, deploying an auto-regressive guardrail model remains challenging: such mod…
RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
Huihao Jing, Wenbin Hu, Shaojin Chen +6
Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Sc…
Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization
Huihao Jing, Shaojin Chen, Wenbin Hu +5
As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem. Guardrails are…
Towards In-Parameter Memory Augmentation for Large Language Models
Haoyu Huang, Zhongwei Xie, Jiaxin Bai +6
Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, a…
SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown
Yuxuan Liu, Haoran Li, Yuhao Zhang +10
Executable Agent Skills combine natural-language instructions and scripts into reusable packages for LLM agents, and revising them requires fixing errors without breaking correct b…
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
Wenbin Hu, Huihao Jing, Haochen Shi +3
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of th…
SoFT: Soft Targets for Generalizable LLM Fine-Tuning
Huihao Jing, Wenbin Hu, Shaojin Chen +7
Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a…
AuthorityLens: Rethinking LLM-Based Agent Systems Through the Lens of Authority
Shaojin Chen, Huihao Jing, Wun Yu Chan +7
LLM-based agents are increasingly deployed with authority over consequential resources and decisions in real systems. These agents often operate alongside human and LLM-based parti…
Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
Hongyu Luo, He Wang, Huihao Jing +6
Current evaluations do not isolate whether text-only language models can originate visual concepts before image generation. Fluent visual prose can hide visual-plan failures: an an…
Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds
Yuxuan Liu, Zhaochen Su, Yuhao Zhang +9
Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model. Yet it remains unclear when…
RLPF: Reinforcement Learning from Performance Feedback for Code Generation
Huihao Jing, Haozhe Cui, Wenbin Hu +9
Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can p…
Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions
Huihao Jing, Wenbin Hu, Shaojin Chen +10
The capability of LLM agents to function as the ``brain'' of a system fundamentally expands the scope of analysis beyond a standalone model. Consequently, safety is no longer only…
SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents
Qiao Xiao, Haochen Shi, Yisen Gao +9
Large language model (LLM) agents increasingly rely on agent harnesses that manage context, tools, and multi-turn execution, making tools a central interface for acting in realisti…
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
Huihao Jing, Wenbin Hu, Shaojin Chen +5
Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains lim…
DeepRefine: Agentic Knowledge Refinement via Reinforcement Learning
Haoyu Huang, Jiaxin Bai, Shujie Liu +7
External knowledge enables large language model (LLM) agents to ground their actions and decisions beyond intrinsic parametric memory in open-ended, knowledge-intensive downstream…
OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset
Wenbin Hu, Huihao Jing, Haochen Shi +3
Ensuring the safety and compliance of large language models (LLMs) is of paramount importance. However, existing LLM safety datasets often rely on ad-hoc taxonomies for data genera…
ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance
Haoran Li, Yulin Chen, Huihao Jing +6
Individuals' concerns about data privacy and AI safety are highly contextualized and extend beyond sensitive patterns. Addressing these issues requires reasoning about the context…
GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety
Changxuan Fan, Xi Yang, Yueyuan Zheng +9
As older adults increasingly use LLM-based chatbots for companionship and assistance, a safety gap is emerging. Older adults may face vulnerabilities from social isolation, limited…
Do Reasoning Models Enhance Embedding Models?
Wun Yu Chan, Shaojin Chen, Huihao Jing +5
State-of-the-art embedding models are increasingly derived from decoder-only Large Language Model (LLM) backbones adapted via contrastive learning. Given the emergence of reasoning…
MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoning
Huihao Jing, Wenbin Hu, Hongyu Luo +4
Multi-agent systems (MAS), leveraging the remarkable capabilities of Large Language Models (LLMs), show great potential in addressing complex tasks. In this context, integrating MA…
Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
Wenbin Hu, Huihao Jing, Haochen Shi +2
The proliferation of Large Language Models (LLMs) has demonstrated remarkable capabilities, elevating the critical importance of LLM safety. However, existing safety methods rely o…
GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
Haoran Li, Jingru Zeng, Yulin Chen +8
As large language models (LLMs) are integrated into numerous applications, LLMs' safety becomes critical for both application developers and intended users. Currently, great effort…