21 citations · 69 across the 34 of their papers we have counts for
35 papers · 1 filter
ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation
Joseph Marvin Imperial, Junhong Liang, Belal Shoer +9
When a text is translated, does the translation retain the complexity of the original? We introduce ComplexityMT, a new challenge for assessing how text complexity and machine tran…
Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt
Claire Bonial, Claire Benet Post, Laura Michaelis +1
Usage-based theories of grammar explain language users' ability to interpret and produce novel combinations of words through two complementary inferential strategies. In cases of c…
Safer Policy Compliance with Dynamic Epistemic Fallback
Joseph Marvin Imperial, Harish Tayyar Madabushi
Humans develop a series of cognitive defenses, known as epistemic vigilance, to combat risks of deception and misinformation from everyday interactions. Developing safeguards for L…
Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces
Joseph Marvin Imperial, Harish Tayyar Madabushi
Policy compliance assessment is a fundamental task of evaluating whether an input case strictly complies with a set of human-defined rules, more generally known as policies. In pra…
Evaluating CxG Generalisation in LLMs via Construction-Based NLI Fine Tuning
Tom Mackintosh, Harish Tayyar Madabushi, Claire Bonial
We probe large language models' ability to learn deep form-meaning mappings as defined by construction grammars. We introduce the ConTest-NLI benchmark of 80k sentences covering ei…
Dancing with Deer: A Constructional Perspective on MWEs in the Era of LLMs
Claire Bonial, Julia Bonn, Harish Tayyar Madabushi
In this chapter, we argue for the benefits of understanding multiword expressions from the perspective of usage-based, construction grammar approaches. We begin with a historical o…
UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment
Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden +16
We introduce UniversalCEFR, a large-scale multilingual and multidimensional dataset of texts annotated with CEFR (Common European Framework of Reference) levels in 13 languages. To…
Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors
Harish Tayyar Madabushi, Melissa Torgbi, Claire Bonial
This position paper argues that many behaviours framed as either "stochastic parroting" or "advanced reasoning" are better explained as context-directed extrapolation from training…
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
Jingcheng Niu, Subhabrata Dutta, Ahmed Elshabrawy +2
Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mech…
Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts
Frances Laureano De Leon, Harish Tayyar Madabushi, Mark G. Lee
Multiword expressions, characterised by non-compositional meanings and syntactic irregularities, are an example of nuanced language. These expressions can be used literally or idio…
The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities
Irina Bigoulaeva, Harish Tayyar Madabushi, Iryna Gurevych
Large Language Models (LLMs), trained on extensive web-scale corpora, have demonstrated remarkable abilities across diverse tasks, especially as they are scaled up. Nevertheless, e…
Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the United Kingdom
Melissa Torgbi, Andrew Clayman, Jordan J. Speight +1
We collect novel data in the public service domain to evaluate the capability of the state-of-the-art automatic speech recognition (ASR) models in capturing regional differences in…
Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions
Wesley Scivetti, Melissa Torgbi, Austin Blodgett +4
The web-scale of pretraining data has created an important evaluation challenge: to disentangle linguistic competence on cases well-represented in pretraining data from generalizat…
SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning
Joseph Marvin Imperial, Harish Tayyar Madabushi
Specialized lexicons are collections of words with associated constraints such as special definitions, specific roles, and intended target audiences. These constraints are necessar…
Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
Haritz Puerto, Tilek Chubakov, Xiaodan Zhu +2
Requiring a large language model (LLM) to generate intermediary reasoning steps, known as Chain of Thought (CoT), has been shown to be an effective way of boosting performance. Pre…
FS-RAG: A Frame Semantics Based Approach for Improved Factual Accuracy in Large Language Models
Harish Tayyar Madabushi
We present a novel extension to Retrieval Augmented Generation with the goal of mitigating factual inaccuracies in the output of large language models. Specifically, our method dra…
Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text
Frances A. Laureano De Leon, Harish Tayyar Madabushi, Mark Lee
Code-switching is a prevalent linguistic phenomenon in which multilingual individuals seamlessly alternate between languages. Despite its widespread use online and recent research…
Standardize: Aligning Language Models with Expert-Defined Standards for Content Generation
Joseph Marvin Imperial, Gail Forey, Harish Tayyar Madabushi
Domain experts across engineering, healthcare, and education follow strict standards for producing quality content such as technical manuals, medication instructions, and children'…
Word Boundary Information Isn't Useful for Encoder Language Models
Edward Gow-Smith, Dylan Phelps, Harish Tayyar Madabushi +2
All existing transformer-based approaches to NLP using subword tokenisation algorithms encode whitespace (word boundary information) through the use of special space symbols (such…
Construction Grammar and Language Models
Harish Tayyar Madabushi, Laurence Romain, Petar Milin +1
Recent progress in deep learning and natural language processing has given rise to powerful models that are primarily trained on a cloze-like task and show some evidence of having…
Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models
Joseph Marvin Imperial, Harish Tayyar Madabushi
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educator…
Are Emergent Abilities in Large Language Models just In-Context Learning?
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva +2
Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been speci…
Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5
Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi +2
We compare sequential fine-tuning with a model for multi-task learning in the context where we are interested in boosting performance on two tasks, one of which depends on the othe…
Sample Efficient Approaches for Idiomaticity Detection
Dylan Phelps, Xuan-Rui Fan, Edward Gow-Smith +3
Deep neural models, in particular Transformer-based pre-trained language models, require a significant amount of data to train. This need for data tends to lead to problems when de…