5 citations · 12 across the 18 of their papers we have counts for
19 papers
Scoring Both Directions: LLMs realize the MRS they cannot reliably parse
Soham Dan
The English Resource Grammar (ERG) is a hand-written computational grammar of English. Given a sentence, its processor, ACE, produces a formal meaning representation called Minimal…
READY or Not: Reliable Enterprise Agent Deployment
Veronica Chatrath, Bryan Zhu, Jingxuan Fan +15
An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, w…
CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
Veronica Chatrath, Bryan Zhu, George Pu +16
Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, lo…
FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
Soham Dan
We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 coll…
A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Models
Soham Dan, Himanshu Beniwal, Thomas Hartvigsen
Large language models (LLMs) are increasingly deployed across languages, but their safety behavior remains uneven across linguistic and cultural contexts. This survey synthesizes w…
LAUDE: LLM-Assisted Unit Test Generation and Debugging of Hardware DEsigns
Deeksha Nandal, Riccardo Revalor, Soham Dan +1
Unit tests are critical in the hardware design lifecycle to ensure that component design modules are functionally correct and conform to the specification before they are integrate…
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
Amey Hengle, Prasoon Bajpai, Soham Dan +1
Existing multilingual long-context benchmarks, often based on the popular needle-in-a-haystack test, primarily evaluate a model's ability to locate specific information buried with…
Are LLMs Ready for Practical Adoption for Assertion Generation?
Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan +1
Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, i.e., detecti…
Large Language Models can be Strong Self-Detoxifiers
Ching-Yun Ko, Pin-Yu Chen, Payel Das +6
Reducing the likelihood of generating harmful and toxic output is an essential task when aligning large language models (LLMs). Existing methods mainly rely on training an external…
Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models
Amey Hengle, Prasoon Bajpai, Soham Dan +1
While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unex…
Needle in the Haystack for Memory Based Large Language Models
Elliot Nelson, Georgios Kollias, Payel Das +2
Current large language models (LLMs) often perform poorly on simple fact retrieval tasks. Here we investigate if coupling a dynamically adaptable external memory to a LLM can allev…
CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design
Nafis Neehal, Bowen Wang, Shayom Debopadhaya +4
CTBench is introduced as a benchmark to assess language models (LMs) in aiding clinical study design. Given study-specific metadata, CTBench evaluates AI models' ability to determi…
AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation
Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan +1
Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, \ie, detectio…
Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis
Kushal Raj Bhandari, Sixue Xing, Soham Dan +1
Large Language Models (LLMs), already shown to ace various unstructured text comprehension tasks, have also remarkably been shown to tackle table (structured) comprehension tasks w…
Large Language Model Confidence Estimation via Black-Box Access
Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh +2
Estimating uncertainty or confidence in the responses of a model can be significant in evaluating trust not only in the responses, but also in the model as a whole. In this paper,…
On the Effects of Fine-tuning Language Models for Text-Based Reinforcement Learning
Mauricio Gruppi, Soham Dan, Keerthiram Murugesan +1
Text-based reinforcement learning involves an agent interacting with a fictional environment using observed text and admissible actions in natural language to complete a task. Prev…
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss +4
Transformer-based Language Models have become ubiquitous in Natural Language Processing (NLP) due to their impressive performance on various tasks. However, expensive training as w…
Language Guided Exploration for RL Agents in Text Environments
Hitesh Golchha, Sahil Yerawar, Dhruvesh Patel +2
Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{ta…
Larimar: Large Language Models with Episodic Memory Control
Payel Das, Subhajit Chaudhury, Elliot Nelson +10
Efficient and accurate updating of knowledge stored in Large Language Models (LLMs) is one of the most pressing research challenges today. This paper presents Larimar - a novel, br…