565 citations · 621 across the 33 of their papers we have counts for
36 papers
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti +2
When using an LLM agent in a consequential domain, making an informed decision about whether to trust its output or intervene requires calibrated confidence in the agent's success.…
MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators
Jackson Hassell, Farima Fatahi Bayat, Pouya Pezeshkpour +1
Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to incidental changes in the ev…
Reflective Prompt Tuning through Language Model Function-Calling
Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour +1
Large language models (LLMs) have become increasingly capable of following instructions and complex reasoning, making prompting a flexible interface for adapting models without par…
AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs
Pouya Pezeshkpour, Estevam Hruschka
Verification is becoming central to both reinforcement-learning-based training and inference-time control of large language models (LLMs). Yet current verifiers face a fundamental…
Blue Data Intelligence Layer: Streaming Data and Agents for Multi-source Multi-modal Data-Centric Applications
Moin Aminnaseri, Farima Fatahi Bayat, Nikita Bhutani +17
NL2SQL systems aim to address the growing need for natural language interaction with data. However, real-world information rarely maps to a single SQL query because (1) users expre…
Align Then Adapt: Label-Efficient Adapter Learning for Asymmetric Dense Retrieval
Seiji Maekawa, Moin Aminnaseri, Pouya Pezeshkpour +1
Dense retrieval systems increasingly face an asymmetry between complex instruction-like queries and relatively simple, static document collections. While stronger embedders can bet…
From Task Solving to Robust Real-World Adaptation in LLM Agents
Pouya Pezeshkpour, Estevam Hruschka
Large language models are increasingly deployed as specialized agents that plan, call tools, and take actions over extended horizons. Yet many existing evaluations assume a "clean…
Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models
Arash Gholami Davoodi, Navid Rezazadeh, Seyed Pouyan Mousavi Davoudi +1
Large language models (LLMs) must balance diversity and creativity against logical coherence in open-ended generation. Existing truncation-based samplers are effective but largely…
From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
Farima Fatahi Bayat, Pouya Pezeshkpour, Estevam Hruschka
Tool-augmented Language Models (TaLMs) can invoke external tools to solve problems beyond their parametric capacity. However, it remains unclear whether these tool-enabled gains re…
Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour +2
Existing benchmarks for tool-augmented language models (TaLMs) lack fine-grained control over task difficulty and remain vulnerable to data contamination. We present FuncBenchGen,…
Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
Pouya Pezeshkpour, Moin Aminnaseri, Estevam Hruschka
Vision-language models (VLMs) have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. However, it is not clear ho…
Insight-RAG: Enhancing LLMs with Insight-Driven Augmentation
Pouya Pezeshkpour, Estevam Hruschka
Retrieval Augmented Generation (RAG) frameworks have shown significant promise in leveraging external knowledge to enhance the performance of large language models (LLMs). However,…
Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Education
Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani +1
Large Language Models (LLMs) offer the potential to automate hiring by matching job descriptions with candidate resumes, streamlining recruitment processes, and reducing operationa…
Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?
Pouya Pezeshkpour, Estevam Hruschka
Large Language Models (LLMs) have demonstrated remarkable performance on various tasks, yet their ability to extract and internalize deeper insights from domain-specific datasets r…
From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization
Catarina G. Belem, Pouya Pezeshkpour, Hayate Iso +3
Although many studies have investigated and reduced hallucinations in large language models (LLMs) for single-document tasks, research on hallucination in multi-document summarizat…
A Blueprint Architecture of Compound AI Systems for Enterprise
Eser Kandogan, Sajjadur Rahman, Nikita Bhutani +11
Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goa…
LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
Arash Gholami Davoodi, Seyed Pouyan Mousavi Davoudi, Pouya Pezeshkpour
Large language models (LLMs) demonstrate impressive capabilities in mathematical reasoning. However, despite these achievements, current evaluations are mostly limited to specific…
Multi-Conditional Ranking with Large Language Models
Pouya Pezeshkpour, Estevam Hruschka
Utilizing large language models (LLMs) to rank a set of items has become a common approach in recommendation and retrieval systems. Typically, these systems focus on ordering a sub…
Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions
Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani +3
Remarkable performance of large language models (LLMs) in a variety of tasks brings forth many opportunities as well as challenges of utilizing them in production settings. Towards…
Distilling Large Language Models using Skill-Occupation Graph Context for HR-Related Tasks
Pouya Pezeshkpour, Hayate Iso, Thom Lake +2
Numerous HR applications are centered around resumes and job descriptions. While they can benefit from advancements in NLP, particularly large language models, their real-world ado…
Less is More for Long Document Summary Evaluation by LLMs
Yunshu Wu, Hayate Iso, Pouya Pezeshkpour +2
Large Language Models (LLMs) have shown promising performance in summary evaluation tasks, yet they face challenges such as high computational costs and the Lost-in-the-Middle prob…
Rethinking Language Models as Symbolic Knowledge Graphs
Vishwas Mruthyunjaya, Pouya Pezeshkpour, Estevam Hruschka +1
Symbolic knowledge graphs (KGs) play a pivotal role in knowledge-centric applications such as search, question answering and recommendation. As contemporary language models (LMs) t…
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pouya Pezeshkpour, Estevam Hruschka
Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks. However, previous works have shown these models are sensitive towards prompt wording, a…
Measuring and Modifying Factual Knowledge in Large Language Models
Pouya Pezeshkpour
Large Language Models (LLMs) store an extensive amount of factual knowledge obtained from vast collections of text. To effectively utilize these models for downstream tasks, it is…