1 citations · 1 across the 12 of their papers we have counts for
13 papers
ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code
Jiaxiang Tang, Yi Zhou, Chad DeLuca +5
Chart editing requires cross-modal edit grounding, realizing a requested visual change in the code that draws it, with necessary related updates and without altering unrelated cont…
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Shailja Thakur, Sungeun An, Chad DeLuca +1
A benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it d…
Runtime-Structured Task Decomposition for Agentic Coding Systems
Shubhi Asthana, Bing Zhang, Chad DeLuca +2
Agentic coding systems increasingly use large language models (LLMs) for software engineering tasks such as debugging, root cause analysis, and code review. However, many existing…
A Systematic Approach for Large Language Models Debugging
Basel Shbita, Anna Lisa Gentile, Bing Zhang +10
Large language models (LLMs) have become central to modern AI workflows, powering applications from open-ended text generation to complex agent-based reasoning. However, debugging…
STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs
Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2
Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill ga…
LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface
Michael Hind, Basel Shbita, Bo Wu +5
Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often…
LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics
Farhan Ahmed, Yuya Jeremy Ong, Chad DeLuca
Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment. However, traditional evaluation approaches provide limited insi…
STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Calls
Shubhi Asthana, Bing Zhang, Chad DeLuca +2
The rapid shift from stateless large language models (LLMs) to autonomous, goal-driven agents raises a central question: When is agentic AI truly necessary? While agents enable mul…
MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation
Basel Shbita, Farhan Ahmed, Chad DeLuca
Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engi…
Backprompting: Leveraging Synthetic Production Data for Health Advice Guardrails
Kellen Tan Cheng, Anna Lisa Gentile, Chad DeLuca +1
The pervasiveness of large language models (LLMs) in enterprise settings has also brought forth a significant amount of risks associated with their usage. Guardrails technologies a…
OneShield -- the Next Generation of LLM Guardrails
Chad DeLuca, Anna Lisa Gentile, Shubhi Asthana +7
The rise of Large Language Models has created a general excitement about the great potential for a myriad of applications. While LLMs offer many possibilities, questions about safe…
Deploying Privacy Guardrails for LLMs: A Comparative Analysis of Real-World Applications
Shubhi Asthana, Bing Zhang, Ruchi Mahindru +3
The adoption of Large Language Models (LLMs) has revolutionized AI applications but poses significant challenges in safeguarding user privacy. Ensuring compliance with privacy regu…
SAUCE: Truncated Sparse Document Signature Bit-Vectors for Fast Web-Scale Corpus Expansion
Muntasir Wahed, Daniel Gruhl, Alfredo Alba +5
Recent advances in text representation have shown that training on large amounts of text is crucial for natural language understanding. However, models trained without predefined n…