9 papers
Operationalising the Superficial Alignment Hypothesis via Task Complexity
Tomás Vergara-Browne, Darshan Patil, Ivan Titov +3
The superficial alignment hypothesis (SAH) posits that large language models learn most of their knowledge during pre-training, and that post-training merely surfaces this knowledg…
BRIDGE: Predicting Human Task Completion Time From Model Performance
Fengyuan Liu, Jay Gala, Nilaksh +3
Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task difficulty. Existing approaches that rely on d…
Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Models
Mridul Sharma, Adeetya Patel, Zaneta D' Souza +3
Large language models are increasingly used for clinical text classification, where overconfident misclassifications can directly affect patient care. Existing black-box uncertaint…
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning
Le Zhang, Bo Wang, Xipeng Qiu +2
We present REARANK, a large language model (LLM)-based listwise reasoning reranking agent. REARANK explicitly reasons before reranking, significantly improving both performance and…
How to Get Your LLM to Generate Challenging Problems for Evaluation
Arkil Patel, Siva Reddy, Dzmitry Bahdanau
The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impractica…
ROSA: Random Subspace Adaptation for Efficient Fine-Tuning
Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy +1
Model training requires significantly more memory, compared with inference. Parameter efficient fine-tuning (PEFT) methods provide a means of adapting large models to downstream ta…
A Compositional Typed Semantics for Universal Dependencies
Laurestine Bradford, Timothy John O'Donnell, Siva Reddy
Languages may encode similar meanings using different sentence structures. This makes it a challenge to provide a single set of formal rules that can derive meanings from sentences…
When does word order matter and when doesn't it?
Xuanda Chen, Timothy O'Donnell, Siva Reddy
Language models (LMs) may appear insensitive to word order changes in natural language understanding (NLU) tasks. In this paper, we propose that linguistic redundancy can explain t…
Evaluating In-Context Learning of Libraries for Code Generation
Arkil Patel, Siva Reddy, Dzmitry Bahdanau +1
Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising area is their ability to interpret code mo…