16 papers · 1 filter
The Backdrop Exposes What the World Around an Agent Costs It
Nusrat Jahan Lia, Shubhashis Roy Dipta
Agent benchmarks test agents in worlds that stay still. Deployed agents work in worlds that other people also change. Someone texts the agent to send the money elsewhere or an orde…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms
Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta +2
Bengali is the seventh-most-spoken language globally, yet LLM safety evaluation remains overwhelmingly English-centric. We introduce BanglaSafe, a benchmark of 879 Bengali prompts…
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta +2
While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical tokens disproportionately drives reasoning gains, an analogous…
Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models
Elias Hossain, Sourav Saha, Tasfia Nuzhat Ornee +5
Language models may encode both outdated facts and their newer replacements. We introduce Parametric Temporal Conflict (PTC), where the newer fact is present and recoverable, but t…
DecomposeRL: Learning to Ask Useful, Informative, and Diverse Questions for Semi-Supervised, Traceable Claim Verification
Shubhashis Roy Dipta, Ankur Padia, Francis Ferraro
Claim verification splits between end-to-end classifiers that are accurate but yields no inspectable traces, and decomposition-based methods produce inspectable traces but lag perf…
TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
Zabir Al Nazi, Shubhashis Roy Dipta
Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to at…
AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators
Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia +10
Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may…
Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan Parvez
Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet une…
UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformers
Elias Hossain, Shubhashis Roy Dipta, Subash Neupane +4
Neural NLP models are often miscalibrated and overconfident, assigning high confidence to incorrect predictions and failing to express uncertainty during internal evidence aggregat…
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6
Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To a…
PA3: Policy-Aware Agent Alignment through Chain-of-Thought
Shubhashis Roy Dipta, Daniel Bis, Kun Zhou +4
Conversational assistants powered by large language models (LLMs) excel at tool-use tasks but struggle with adhering to complex, business-specific rules. While models can reason ov…
BanglaIPA: Towards Robust Text-to-IPA Transcription with Contextual Rewriting in Bengali
Jakir Hasan, Shrestha Datta, Md Saiful Islam +2
Despite its widespread use, Bengali lacks a robust automated International Phonetic Alphabet (IPA) transcription system that effectively supports both standard language and regiona…
Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability
Nusrat Jahan Lia, Shubhashis Roy Dipta
Recent advances in multilingual representation learning aim to bridge the performance gap between high- and low-resource languages, yet their ability to preserve affective meaning…
DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems
Zabir Al Nazi, Shubhashis Roy Dipta, Sudipta Kar
Chain-of-Thought (CoT) prompting is widely adopted for mathematical problem solving, including in low-resource languages, yet its behavior under irrelevant context remains underexp…
GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO
Shubhashis Roy Dipta, Khairul Mahbub, Nadia Najjar
We present a Bengali mathematical reasoning model called GanitLLM (named after the Bangla word for mathematics, Ganit), together with a new difficulty-aware Bengali math corpus and…