72 citations · 116 across the 17 of their papers we have counts for
17 papers
Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning
Kevin Du, Alexander Hoyle, Laura Ruis +1
Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer. A growing body of work treats them as such, using LLM judges…
Position: It's Time to Optimize LLMs for Self-Consistency
Itamar Pres, Belinda Z. Li, Laura Ruis +6
Despite ever-increasing sophistication in language model (LM) pre- and post-training pipelines, many important failures persist: models overcondition on user framing ("sycophancy")…
Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision
Zifan Carl Guo, Laura Ruis, Jacob Andreas +1
When does training language models (LMs) to generate explanations of their predictions yield faithful introspection, rather than superficial imitation? We study LMs trained to expl…
Self-CTRL: Self-Consistency Training with Reinforcement Learning
Itamar Pres, Laura Ruis, Melat Ghebreselassie +2
Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with…
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Jing Huang, Daniel Wurgaft, Rachit Bansal +6
Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger mo…
The Depth Ceiling: On the Limits of Large Language Models in Discovering Latent Planning
Yi Xu, Philipp Jettkant, Laura Ruis
The viability of chain-of-thought (CoT) monitoring hinges on models being unable to reason effectively in their latent representations. Yet little is known about the limits of such…
Infusion: Shaping Model Behavior by Editing Training Data via Influence Functions
J Rosser, Robert Kirk, Edward Grefenstette +2
Influence functions are commonly used to attribute model behavior to training documents. We explore the reverse: crafting training data that induces model behavior. Our framework,…
Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs
Jonathan Cook, Silvia Sapora, Arash Ahmadian +4
Large language models (LLMs) are typically trained to acquire behaviours from demonstrations or experience, yet much of their training data is declarative: instructions, rules, and…
Command A: An Enterprise-Ready Large Language Model
Team Cohere, :, Aakanksha +227
In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command A is an agent-optimised…
Investigating Non-Transitivity in LLM-as-a-Judge
Yi Xu, Laura Ruis, Tim Rocktäschel +1
Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most…
Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
Laura Ruis, Maximilian Mozes, Juhan Bae +7
The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand,…
Towards Reliable Evaluation of Behavior Steering Interventions in LLMs
Itamar Pres, Laura Ruis, Ekdeep Singh Lubana +1
Representation engineering methods have recently shown promise for enabling efficient steering of model behavior. However, evaluation pipelines for these methods have primarily rel…
Debating with More Persuasive LLMs Leads to More Truthful Answers
Akbir Khan, John Hughes, Dan Valentine +7
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will s…
The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Laura Ruis, Akbir Khan, Stella Biderman +3
Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context -- incorpora…
Improving Systematic Generalization Through Modularity and Augmentation
Laura Ruis, Brenden Lake
Systematic generalization is the ability to combine known parts into novel meaning; an important aspect of efficient human learning, but a weakness of neural network learning. In t…
A Benchmark for Systematic Generalization in Grounded Language Understanding
Laura Ruis, Jacob Andreas, Marco Baroni +2
Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by…
Insertion-Deletion Transformer
Laura Ruis, Mitchell Stern, Julia Proskurnia +1
We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are…