4 papers
Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges
Hyo Jin Do, Zahra Ashktorab, Jasmina Gajcin +5
The LLM-as-a-judge paradigm enables flexible, user-defined evaluation, but its effectiveness is often limited by the scarcity of diverse, representative data for refining criteria.…
EvalAssist: A Human-Centered Tool for LLM-as-a-Judge
Zahra Ashktorab, Werner Geyer, Michael Desmond +6
With the broad availability of large language models and their ability to generate vast outputs using varied prompts and configurations, determining the best output for a given tas…
Agentic AI Needs a Systems Theory
Erik Miehling, Karthikeyan Natesan Ramamurthy, Kush R. Varshney +11
The endowment of AI with reasoning capabilities and some degree of agency is widely viewed as a path toward more capable and generalizable systems. Our position is that the current…
Granite Guardian
Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia +20
We introduce the Granite Guardian models, a suite of safeguards designed to provide risk detection for prompts and responses, enabling safe and responsible use in combination with…