5 citations · 5 across the 4 of their papers we have counts for
5 papers
RoleFUZZER: User-Behaviour Fuzzing for Role-Specification Adherence in Large Language Models
Kshitish Ghate, Michael Saxon, Tadayoshi Kohno +1
Large language models (LLMs) are often conditioned into specific roles such as tutors, interviewers, or agents acting for an organisation, through system prompts that specify how t…
RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs
Damian Hodel, Jevin West, Aylin Caliskan
Language models (LMs) exhibit problematic biases, such as stereotypes. Effectively analyzing and mitigating such biases requires accurate and generalizable evaluation methods of th…
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3
Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are unclear, impatient, or reluctant to share in…
Deep Reasoning in General Purpose Agents via Structured Meta-Cognition
Dean Light, Michael Theologitis, Kshitish Ghate +7
Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambiguity through associative judgment,…
ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor
Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage +1
How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Lib…