FlauBERT: Unsupervised Language Model Pre-training for French

Abstract

Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.

Accepted to LREC 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• CamemBERT: a Tasty French Language Model
• AraBERT: Transformer-based Model for Arabic Language Understanding
• Semi-supervised Sequence Learning
• Reducing Transformer Depth on Demand with Structured Dropout
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• BERTje: A Dutch BERT Model
• Portuguese Named Entity Recognition using BERT-CRF
• XNLI: Evaluating Cross-lingual Sentence Representations
• Multilingual is not enough: BERT for Finnish
• Fixup Initialization: Residual Learning Without Normalization

Cited by in corpus (6)

• Pre-trained Models for Natural Language Processing: A Survey
• Sabiá: Portuguese Large Language Models
• Latin BERT: A Contextual Language Model for Classical Philology
• Ensembling Transformers for Cross-domain Automatic Term Extraction
• Czert -- Czech BERT-like Model for Language Representation
• HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics