Publications (56)
Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
Manuel Faysse, Gautier Viaud, Céline Hudelot +1
Instruction Fine-Tuning (IFT) is a powerful paradigm that strengthens the zero-shot capabilities of Large Language Models (LLMs), but in doing so induces new evaluation metric requ…
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Duarte M. Alves, José Pombal, Nuno M. Guerreiro +10
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only wh…
ModernVBERT: Towards Smaller Visual Document Retrievers
Paul Teiletche, Quentin Macé, Max Conti +4
Retrieving specific information from a large corpus of documents is a prevalent industrial use case of modern AI, notably due to the popularity of Retrieval-Augmented Generation (R…
Does It Tie Out? Towards Autonomous Legal Agents in Venture Capital
Pierre Colombo, Malik Boudiaf, Allyn Sweet +4
Before closing venture capital financing rounds, lawyers conduct diligence that includes tying out the capitalization table: verifying that every security (for example, shares, opt…
ColPali: Efficient Document Retrieval with Vision Language Models
Manuel Faysse, Hugues Sibille, Tony Wu +4
Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on th…
Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data
Maxime Darrin, Pablo Piantanida, Pierre Colombo
Implementing effective control mechanisms to ensure the proper functioning and security of deployed NLP models, from translation to chatbots, is essential. A key ingredient to ensu…
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Hugo Laurençon, Lucile Saulnier, Thomas Wang +51
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop,…
Code-switched inspired losses for generic spoken dialog representations
Emile Chapuis, Pierre Colombo, Matthieu Labeau +1
Spoken dialog systems need to be able to handle both multiple languages and multilinguality inside a conversation (\textit{e.g} in case of code-switching). In this work, we introdu…
Beam Search with Bidirectional Strategies for Neural Response Generation
Pierre Colombo, Chouchang Yang, Giovanna Varni +1
Sequence-to-sequence neural networks have been widely used in language-based applications as they have flexible capabilities to learn various language models. However, when seeking…
The Glass Ceiling of Automatic Evaluation in Natural Language Generation
Pierre Colombo, Maxime Peyrard, Nathan Noiry +2
Automatic evaluation metrics capable of replacing human judgments are critical to allowing fast development of new methods. Thus, numerous research efforts have focused on crafting…
A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations
Pierre Colombo, Chloe Clavel, Pablo Piantanida
Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others…
EuroBERT: Scaling Multilingual Encoders for European Languages
Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves +16
General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their w…
EuroLLM-9B: Technical Report
Pedro Henrique Martins, João Alves, Patrick Fernandes +14
This report presents EuroLLM-9B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 a…
Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
Pierre Colombo, Victor Pellegrain, Malik Boudiaf +5
Proprietary and closed APIs are becoming increasingly common to process natural language, and are impacting the practical applications of natural language processing, including few…
Guider l'attention dans les modeles de sequence a sequence pour la prediction des actes de dialogue
Pierre Colombo, Emile Chapuis, Matteo Manica +3
The task of predicting dialog acts (DA) based on conversational dialog is a key component in the development of conversational agents. Accurately predicting DAs requires a precise…
BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef +2
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: th…
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Duarte M. Alves, Nuno M. Guerreiro, João Alves +5
Large language models (LLMs) are a promising avenue for machine translation (MT). However, current LLM-based MT systems are brittle: their effectiveness highly depends on the choic…
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann +122
Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained…
Hallucinations in Large Multilingual Translation Models
Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf +4
Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages, making them increasingly appealing for r…
Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings
Max Conti, Manuel Faysse, Gautier Viaud +3
A limitation of modern document retrieval embedding methods is that they typically encode passages (chunks) from the same documents independently, often overlooking crucial context…
EuroLLM: Multilingual Language Models for Europe
Pedro Henrique Martins, Patrick Fernandes, João Alves +12
The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at develo…
Beyond Mahalanobis-Based Scores for Textual OOD Detection
Pierre Colombo, Eduardo D. C. Gomes, Guillaume Staerman +2
Deep learning methods have boosted the adoption of NLP systems in real-life applications. However, they turn out to be vulnerable to distribution shifts over time which may cause s…
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation
Pierre Colombo, Chloe Clavel, Pablo Piantanida
Assessing the quality of natural language generation systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non…
Automatic Text Evaluation through the Lens of Wasserstein Barycenters
Pierre Colombo, Guillaume Staerman, Chloe Clavel +1
A new metric \texttt{BaryScore} to evaluate text generation based on deep contextualized embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new fram…
SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
Pierre Colombo, Telmo Pires, Malik Boudiaf +7
In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language models (LLMs) tailored for the legal sector. These models, which feature architectures of 54 billion and…
A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
Pierre Colombo, Nathan Noiry, Guillaume Staerman +1
One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More preci…
A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions
Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo +2
The design of a metric between probability distributions is a longstanding problem motivated by numerous applications in Machine Learning. Focusing on continuous probability distri…
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
Cyril Chhun, Pierre Colombo, Chloé Clavel +1
Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no anal…
Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation
Anas Himmi, Guillaume Staerman, Marine Picot +2
Hallucinated translations pose significant threats and safety concerns when it comes to the practical deployment of machine translation systems. Previous research works have identi…
A Functional Data Perspective and Baseline On Multi-Layer Out-of-Distribution Detection
Eduardo Dadalto, Pierre Colombo, Guillaume Staerman +2
A key feature of out-of-distribution (OOD) detection is to exploit a trained neural network by extracting statistical patterns and relationships through the multi-layer classifier…
EuroLLM-22B: Technical Report
Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef +15
This report presents EuroLLM-22B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11…
SaulLM-7B: A pioneering Large Language Model for Law
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf +8
In this paper, we introduce SaulLM-7B, a large language model (LLM) tailored for the legal domain. With 7 billion parameters, SaulLM-7B is the first LLM designed explicitly for leg…
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2
Distilling reasoning traces from strong teacher models has become the standard recipe for building capable small language models. Yet reasoning traces are 5-20 longer than…
Should We Still Pretrain Encoders with Masked Language Modeling?
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse +5
Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent…
Learning Disentangled Textual Representations via Statistical Measures of Similarity
Pierre Colombo, Guillaume Staerman, Nathan Noiry +1
When working with textual data, a natural application of disentangled representations is fair classification where the goal is to make predictions without being biased (or influenc…
Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
Hippolyte Gisserot-Boukhlef, Manuel Faysse, Emmanuel Malherbe +2
Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Information Retrieval (IR) systems. Yet, failures remain frequent, the models used often being un…
Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
Hippolyte Gisserot-Boukhlef, Ricardo Rei, Emmanuel Malherbe +3
Neural metrics for machine translation (MT) evaluation have become increasingly prominent due to their superior correlation with human judgments compared to traditional lexical met…
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Nicolas Boizard, Kevin El Haddad, Céline Hudelot +1
Deploying large language models (LLMs) of several billion parameters can be impractical in most industrial use cases due to constraints such as cost, latency limitations, and hardw…
Guiding attention in Sequence-to-sequence models for Dialogue Act prediction
Pierre Colombo, Emile Chapuis, Matteo Manica +3
The task of predicting dialog acts (DA) based on conversational dialog is a key component in the development of conversational agents. Accurately predicting DAs requires a precise…
CroissantLLM: A Truly Bilingual French-English Language Model
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro +13
We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully op…
Unsupervised Layer-wise Score Aggregation for Textual OOD Detection
Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes +3
Out-of-distribution (OOD) detection is a rapidly growing field due to new robustness and security requirements driven by an increased number of AI-based systems. Existing OOD textu…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
Hierarchical Pre-training for Sequence Labelling in Spoken Dialog
Emile Chapuis, Pierre Colombo, Matteo Manica +2
Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic…
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe +2
Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluati…
xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection
Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt +3
Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level scor…
Heavy-tailed Representations, Text Polarity Classification & Data Augmentation
Hamid Jalalzai, Pierre Colombo, Chloé Clavel +4
The dominant approaches to text representation in natural language rely on learning embeddings on massive corpora which have convenient properties such as compositionality and dist…
From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
Alexandre Garcia, Pierre Colombo, Slim Essid +2
The task of predicting fine grained user opinion based on spontaneous spoken language is a key problem arising in the development of Computational Agents as well as in the developm…
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida +1
Neural machine translation (NMT) has become the de-facto standard in real-world machine translation applications. However, NMT models can unpredictably produce severely pathologica…
Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
Anas Himmi, Ekhine Irurozki, Nathan Noiry +2
The evaluation of natural language processing (NLP) systems is crucial for advancing the field, but current benchmarking approaches often assume that all systems have scores availa…
Affect-Driven Dialog Generation
Pierre Colombo, Wojciech Witon, Ashutosh Modi +2
The majority of current systems for end-to-end dialog generation focus on response quality without an explicit control over the affective content of the responses. In this paper, w…
What are the best systems? New perspectives on NLP Benchmarking
Pierre Colombo, Nathan Noiry, Ekhine Irurozki +1
In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are…
TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
André G. Viveiros, Patrick Fernandes, Saul Santos +7
Despite significant advances in vision-language models (VLMs), most existing work follows an English-centric design process, limiting their effectiveness in multilingual settings.…
A Differential Entropy Estimator for Training Neural Networks
Georg Pichler, Pierre Colombo, Malik Boudiaf +2
Mutual Information (MI) has been widely used as a loss regularizer for training neural networks. This has been particularly effective when learn disentangled or compressed represen…
The importance of fillers for text representations of speech transcripts
Tanvi Dinkar, Pierre Colombo, Matthieu Labeau +1
While being an essential component of spoken language, fillers (e.g."um" or "uh") often remain overlooked in Spoken Language Understanding (SLU) tasks. We explore the possibility o…
Toward Stronger Textual Attack Detectors
Pierre Colombo, Marine Picot, Nathan Noiry +2
The landscape of available textual adversarial attacks keeps growing, posing severe threats and raising concerns regarding the deep NLP system's integrity. However, the crucial pro…
Improving Multimodal fusion via Mutual Dependency Maximisation
Pierre Colombo, Emile Chapuis, Matthieu Labeau +1
Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic. Acknowledging humans communicate through a variety of chann…