14 citations · 33 across the 18 of their papers we have counts for
20 papers
Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
Li Lucy, Myra Cheng, Svenja Guhr +7
The sycophancy of language models, or their failure to push back against faulty user input, is now widely studied. However, if LMs do push back, is sycophancy solved? We show that…
Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtext
Kabir Ahuja, Yuxuan Li, Andrew Kyle Lampinen
Human communication is fundamentally creative, and often makes use of subtext -- implied meaning that goes beyond the literal content of the text. Here, we systematically study whe…
BLAB: Brutally Long Audio Bench
Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja +13
Developing large audio language models (LMs) capable of understanding diverse spoken interactions is essential for accommodating the multimodal nature of human communication and ca…
Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Kabir Ahuja, Melanie Sclar, Yulia Tsvetkov
Stories are a fundamental aspect of human experience. Engaging deeply with stories and spotting plot holes -- inconsistencies in a storyline that break the internal logic or rules…
Teaching Transformers Causal Reasoning through Axiomatic Training
Aniket Vashishtha, Abhinav Kumar, Atharva Pandey +4
For text-based AI systems to interact in the real world, causal reasoning is an essential skill. Since active interventions are costly, we study to what extent a system can learn c…
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar +4
Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures without explicitly en…
DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages
Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava +4
Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is langu…
MAFIA: Multi-Adapter Fused Inclusive LanguAge Models
Prachi Jain, Ashutosh Sathe, Varun Gumma +2
Pretrained Language Models (PLMs) are widely used in NLP for various tasks. Recent studies have identified various biases that such models exhibit and have proposed methods to corr…
On Evaluating and Mitigating Gender Biases in Multilingual Settings
Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram
While understanding and removing gender biases in language models has been a long-standing problem in Natural Language Processing, prior research work has primarily been limited to…
In-Context Learning through the Bayesian Prism
Madhur Panwar, Kabir Ahuja, Navin Goyal
In-context learning (ICL) is one of the surprising and useful features of large language models and subject of intense research. Recently, stylized meta-learning-like ICL setups ha…
Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs
Somnath Kumar, Vaibhav Balloli, Mercy Ranjit +5
Large language models (LLMs) have revolutionized various domains but still struggle with non-Latin scripts and low-resource languages. This paper addresses the critical challenge o…
MEGA: Multilingual Evaluation of Generative AI
Kabir Ahuja, Harshita Diddee, Rishav Hada +9
Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important qu…
On the Calibration of Massively Multilingual Language Models
Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat +1
Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in ev…
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
Kabir Ahuja, Shanu Kumar, Sandipan Dandapat +1
Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from…
Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages
Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram +1
Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data i…
On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data
Kabir Ahuja, Monojit Choudhury, Sandipan Dandapat
Borrowing ideas from {\em Production functions} in micro-economics, in this paper we introduce a framework to systematically evaluate the performance and cost trade-offs between ma…
Global Readiness of Language Technology for Healthcare: What would it Take to Combat the Next Pandemic?
Ishani Mondal, Kabir Ahuja, Mohit Jain +3
The COVID-19 pandemic has brought out both the best and worst of language technology (LT). On one hand, conversational agents for information dissemination and basic diagnosis have…
On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages
Satwik Bhattamishra, Kabir Ahuja, Navin Goyal
While recurrent models have been effective in NLP tasks, their performance on context-free languages (CFLs) has been found to be quite weak. Given that CFLs are believed to capture…
On the Ability and Limitations of Transformers to Recognize Formal Languages
Satwik Bhattamishra, Kabir Ahuja, Navin Goyal
Transformers have supplanted recurrent models in a large number of NLP tasks. However, the differences in their abilities to model different syntactic properties remain largely unk…
Syntax-guided Controlled Generation of Paraphrases
Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli +1
Given a sentence (e.g., "I like mangoes") and a constraint (e.g., sentiment flip), the goal of controlled text generation is to produce a sentence that adapts the input sentence to…