85 citations · 203 across the 27 of their papers we have counts for
30 papers
Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks
Dimitrios Rontogiannis, Maxime Peyrard, Nicolas Baldwin +3
Standard single-turn, static benchmarks fall short in evaluating the nuanced capabilities of Large Language Models (LLMs) on complex tasks such as software engineering. In this wor…
zip2zip: Inference-Time Adaptive Tokenization via Online Compression
Saibo Geng, Nathan Ranchin, Yunzhen yao +4
Tokenization efficiency plays a critical role in the performance and cost of large language models (LLMs), yet most models rely on static tokenizers optimized on general-purpose co…
Agentic AI: The Era of Semantic Decoding
Maxime Peyrard, Martin Josifoski, Robert West
Recent work demonstrated great promise in the idea of orchestrating collaborations between LLMs, human input, and various tools to address the inherent limitations of LLMs. We prop…
Symbolic Autoencoding for Self-Supervised Sequence Learning
Mohammad Hossein Amani, Nicolas Mario Baldwin, Amin Mansouri +3
Traditional language models, adept at next-token prediction in text sequences, often struggle with transduction tasks between distinct symbolic systems, particularly when parallel…
Evaluating Language Model Agency through Negotiations
Tim R. Davidson, Veniamin Veselovsky, Martin Josifoski +4
We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings o…
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Giovanni Monea, Maxime Peyrard, Martin Josifoski +6
Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context. Yet the mechanisms underlying this contextual grounding remain unkno…
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
Marija Šakota, Maxime Peyrard, Robert West
Generative language models (LMs) have become omnipresent across data science. For a wide variety of tasks, inputs can be phrased as natural language prompts for an LM, from whose o…
Flows: Building Blocks of Reasoning and Collaborating AI
Martin Josifoski, Lars Klein, Maxime Peyrard +8
Recent advances in artificial intelligence (AI) have produced highly capable and controllable systems. This creates unprecedented opportunities for structured reasoning as well as…
Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
Saibo Geng, Martin Josifoski, Maxime Peyrard +1
Despite their impressive performance, large language models (LMs) still struggle with reliably generating complex output structures when not finetuned to follow the required output…
REFINER: Reasoning Feedback on Intermediate Representations
Debjit Paul, Mete Ismayilzada, Maxime Peyrard +4
Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e.g., chain-of-thought prompting. However, the…
Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction
Martin Josifoski, Marija Sakota, Maxime Peyrard +1
Large language models (LLMs) have great potential for synthetic data generation. This work shows that useful data can be synthetically generated even for tasks that cannot be solve…
Language Model Decoding as Likelihood-Utility Alignment
Martin Josifoski, Maxime Peyrard, Frano Rajic +8
A critical component of a successful language generation pipeline is the decoding algorithm. However, the general principles that should guide the choice of a decoding algorithm re…
Distribution inference risks: Identifying and mitigating sources of leakage
Valentin Hartmann, Léo Meynent, Maxime Peyrard +3
A large body of work shows that machine learning (ML) models can leak sensitive or confidential information about their training data. Recently, leakage due to distribution inferen…
The Glass Ceiling of Automatic Evaluation in Natural Language Generation
Pierre Colombo, Maxime Peyrard, Nathan Noiry +2
Automatic evaluation metrics capable of replacing human judgments are critical to allowing fast development of new methods. Thus, numerous research efforts have focused on crafting…
Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning
Damien Teney, Maxime Peyrard, Ehsan Abbasnejad
Machine learning (ML) models are typically optimized for their accuracy on a given dataset. However, this predictive criterion rarely captures all desirable properties of a model,…
Descartes: Generating Short Descriptions of Wikipedia Articles
Marija Sakota, Maxime Peyrard, Robert West
Wikipedia is one of the richest knowledge sources on the Web today. In order to facilitate navigating, searching, and maintaining its content, Wikipedia's guidelines state that all…
On the Context-Free Ambiguity of Emoji
Justyna Czestochowska, Kristina Gligoric, Maxime Peyrard +6
Emojis come with prepacked semantics making them great candidates to create new forms of more accessible communications. Yet, little is known about how much of this emojis semantic…
GenIE: Generative Information Extraction
Martin Josifoski, Nicola De Cao, Maxime Peyrard +2
Structured and grounded representation of text is typically formalized by closed information extraction, the problem of extracting an exhaustive set of (subject, relation, object)…
Better than Average: Paired Evaluation of NLP Systems
Maxime Peyrard, Wei Zhao, Steffen Eger +1
Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of aver…
Invariant Language Modeling
Maxime Peyrard, Sarvjeet Singh Ghotra, Martin Josifoski +5
Large pretrained language models are critical components of modern NLP pipelines. Yet, they suffer from spurious correlations, poor out-of-domain generalization, and biases. Inspir…
Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?
Maxime Peyrard, Beatriz Borges, Kristina Gligorić +1
The automatic detection of humor poses a grand challenge for natural language processing. Transformer-based systems have recently achieved remarkable results on this task, but they…
KLearn: Background Knowledge Inference from Summarization Data
Maxime Peyrard, Robert West
The goal of text summarization is to compress documents to the relevant information while excluding background information already known to the receiver. So far, summarization rese…
Experts and authorities receive disproportionate attention on Twitter during the COVID-19 crisis
Kristina Gligorić, Manoel Horta Ribeiro, Martin Müller +5
Timely access to accurate information is crucial during the COVID-19 pandemic. Prompted by key stakeholders' cautioning against an "infodemic", we study information sharing on Twit…
On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
Wei Zhao, Goran Glavaš, Maxime Peyrard +3
Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual simil…