33 citations · 88 across the 25 of their papers we have counts for
28 papers
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Anubhav Jangra, Adrian de Wynter
Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropria…
What Was Said, Not What Was 'Thought': Type-6 Logic for CoT Verification
Adrian de Wynter
We introduce Type-6 logic, a variant of dynamic epistemic logic augmented with two operators (uncertainty and recurrence), designed to model the inferential dynamics of contemporar…
If LLMs Have Human-Like Attributes, Then So Does Age of Empires II
Adrian de Wynter
Much research has been carried out on large language models (LLMs) and LLM-powered agentic workflows. However, many works within the field state emergence of, ascribe to, or assume…
Foundation-Preserving Optimization in Generalized Eigenspace
Dongjun Kim, Adrian de Wynter, Huancheng Chen +2
While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining. Existing forgetting aware…
The Hrunting of AI: Where and How to Improve English Dialectal Fairness
Wei Li, Adrian de Wynter
It is known that large language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality a…
Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
Amartya Roy, Elamparithy M, Kripabandhu Ghosh +2
In context learning (ICL) underpins recent advances in large language models (LLMs), although its role and performance in causal reasoning remains unclear. Causal reasoning demands…
Is In-Context Learning Learning?
Adrian de Wynter
In-context learning (ICL) allows some autoregressive models to solve tasks via next-token prediction and without needing further training. This has led to claims about these model'…
Evaluating Style-Personalized Text Generation: Challenges and Directions
Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2
With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--"write like me"--has become a rapidly growing are…
The Thin Line Between Comprehension and Persuasion in LLMs
Adrian de Wynter, Tangming Yuan
Large language models (LLMs) are excellent at maintaining high-level, convincing dialogue, but it remains unclear whether their persuasive success reflects genuine understanding of…
Algorithmically Establishing Trust in Evaluators
Adrian de Wynter
An evaluator, such as an LLM-as-a-judge, is trustworthy when there exists some agreed-upon way to measure its performance as a labeller. Traditional approaches either rely on testi…
A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications
Sunayana Sitaram, Adrian de Wynter, Isobel McCrum +2
Misgendering is the act of referring to someone by a gender that does not match their chosen identity. It marginalizes and undermines a person's sense of self, causing significant…
If Eleanor Rigby Had Met ChatGPT: A Study on Loneliness in a Post-LLM World
Adrian de Wynter
Warning: this paper discusses content related, but not limited to, violence, sex, and suicide. Loneliness, or the lack of fulfilling relationships, significantly impacts a person's…
Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
Fangru Lin, Shaoguang Mao, Emanuele La Malfa +7
Language is not monolithic. While benchmarks, including those designed for multiple languages, are often used as proxies to evaluate the performance of Large Language Models (LLMs)…
Awes, Laws, and Flaws From Today's LLM Research
Adrian de Wynter
We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works released betwe…
RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?
Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri +30
Large language models (LLMs) and small language models (SLMs) are being adopted at remarkable speed, although their safety still remains a serious concern. With the advent of multi…
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Zhang, Shaoguang Mao, Tao Ge +7
This paper presents a comprehensive survey of the current status and opportunities for Large Language Models (LLMs) in strategic reasoning, a sophisticated form of reasoning that n…
Will GPT-4 Run DOOM?
Adrian de Wynter
We show that GPT-4's reasoning and planning capabilities extend to the 1993 first-person shooter Doom. This large language model (LLM) is able to run and play the game with only a…
On Meta-Prompting
Adrian de Wynter, Xun Wang, Qilong Gu +1
Modern large language models (LLMs) are capable of interpreting input strings as instructions, or prompts, and carry out tasks based on them. Unlike traditional learners, LLMs cann…
"I'd Like to Have an Argument, Please": Argumentative Reasoning in Large Language Models
Adrian de Wynter, Tangming Yuan
We evaluate two large language models (LLMs) ability to perform argumentative reasoning. We experiment with argument mining (AM) and argument pair extraction (APE), and evaluate th…
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Rishav Hada, Varun Gumma, Adrian de Wynter +5
Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top , remains inadequate due t…
A User-Centered Evaluation of Spanish Text Simplification
Adrian de Wynter, Anthony Hevia, Si-Qing Chen
We present an evaluation of text simplification (TS) in Spanish for a production system, by means of two corpora focused in both complex-sentence and complex-word identification. W…
An Evaluation on Large Language Model Outputs: Discourse and Memorization
Adrian de Wynter, Xun Wang, Alex Sokolov +2
We present an empirical evaluation of various outputs generated by nine of the most widely-available large language models (LLMs). Our analysis is done with off-the-shelf, readily-…
Turing Completeness and Sid Meier's Civilization
Adrian de Wynter
We prove that three strategy video games from the Sid Meier's Civilization series: Sid Meier's Civilization: Beyond Earth, Sid Meier's Civilization V, and Sid Meier's Civilization…
Optimal Subarchitecture Extraction For BERT
Adrian de Wynter, Daniel J. Perry
We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture…