53 citations · 97 across the 10 of their papers we have counts for
14 papers
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
Zheng Zhao, Clara Vania, Subhradeep Kayal +3
Large language models (LLMs) have advanced conversational AI assistants. However, systematically evaluating how well these assistants apply personalization--adapting to individual…
WebIE: Faithful and Robust Information Extraction on the Web
Chenxi Whitehouse, Clara Vania, Alham Fikri Aji +2
Extracting structured and grounded fact triples from raw text is a fundamental task in Information Extraction (IE). Existing IE datasets are typically collected from Wikipedia arti…
Endowing Language Models with Multimodal Knowledge Graph Representations
Ningyuan Huang, Yash R. Deshpande, Yibo Liu +4
We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG usin…
IndoNLI: A Natural Language Inference Dataset for Indonesian
Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan +2
We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We adapt the data collection protocol for MNLI and collect nearly 18K sentence pairs annotated by crowd wor…
Comparing Test Sets with Item Response Theory
Clara Vania, Phu Mon Htut, William Huang +6
Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrain…
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
Nikita Nangia, Saku Sugawara, Harsh Trivedi +3
Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understanding…
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
Clara Vania, Ruijie Chen, Samuel R. Bowman
Large-scale natural language inference (NLI) datasets such as SNLI or MNLI have been created by asking crowdworkers to read a premise and write three new hypotheses, one for each p…
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nikita Nangia, Clara Vania, Rasika Bhalerao +1
Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases tha…
VisualSem: A High-quality Knowledge Graph for Vision and Language
Houda Alberts, Teresa Huang, Yash Deshpande +4
An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge…
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
Yada Pruksachatkun, Jason Phang, Haokun Liu +6
While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-r…
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Jason Phang, Iacer Calixto, Phu Mon Htut +5
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on…
LINSPECTOR: Multilingual Probing Tasks for Word Representations
Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov +1
Despite an ever growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what i…
What do character-level models learn about morphology? The case of dependency parsing
Clara Vania, Andreas Grivas, Adam Lopez
When parsing morphologically-rich languages with neural models, it is beneficial to model input at the character level, and it has been claimed that this is because character-level…
From Characters to Words to in Between: Do We Capture Morphology?
Clara Vania, Adam Lopez
Words can be represented by composing the representations of subword units such as word segments, characters, and/or character n-grams. While such representations are effective and…