CamemBERT: a Tasty French Language Model

Abstract

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

ACL 2020 long paper. Web site: https://camembert-model.fr

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Cross-lingual Language Model Pretraining
• CamemBERT: a Tasty French Language Model
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• How multilingual is Multilingual BERT?
• Multilingual is not enough: BERT for Finnish
• What the [MASK]? Making Sense of Language-Specific BERT Models
• Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
• FQuAD: French Question Answering Dataset

Cited by in corpus (101)

• Pre-trained Models for Natural Language Processing: A Survey
• CamemBERT: a Tasty French Language Model
• AraBERT: Transformer-based Model for Arabic Language Understanding
• mT5: A massively multilingual pre-trained text-to-text transformer
• Big Bird: Transformers for Longer Sequences
• IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
• Named Entity Recognition and Classification on Historical Documents: A Survey
• GREEK-BERT: The Greeks visiting Sesame Street
• What the [MASK]? Making Sense of Language-Specific BERT Models
• Sabiá: Portuguese Large Language Models
• FlauBERT: Unsupervised Language Model Pre-training for French
• What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models
• On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition
• MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain
• Traditional Machine Learning Models and Bidirectional Encoder Representations From Transformer (BERT)-Based Automatic Classification of Tweets About Eating Disorders: Algorithm Development and Validation Study
• Bringing order into the realm of Transformer-based language models for artificial intelligence and law
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
• Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*
• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
• Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classification
• WikiBERT models: deep transfer learning for many languages
• Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
• KR-BERT: A Small-Scale Korean-Specific Language Model
• Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
• FQuAD: French Question Answering Dataset
• HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
• When Do You Need Billions of Words of Pretraining Data?
• Ensembling Transformers for Cross-domain Automatic Term Extraction
• Large Scale Genealogical Information Extraction From Handwritten Quebec Parish Records
• Empowering machine learning models with contextual knowledge for enhancing the detection of eating disorders in social media posts
• FinEst BERT and CroSloEngual BERT: less is more in multilingual models
• Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation
• Topic Modeling with Contextualized Word Representation Clusters
• Zero-shot Bilingual App Reviews Mining with Large Language Models
• Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• WangchanBERTa: Pretraining transformer-based Thai Language Models
• German's Next Language Model
• Give your Text Representation Models some Love: the Case for Basque
• Czert -- Czech BERT-like Model for Language Representation
• Modern French Poetry Generation with RoBERTa and GPT-2
• Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yorùbá and Twi
• Large Scale Multimodal Classification Using an Ensemble of Transformer Models and Co-Attention
• RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning
• DziriBERT: a Pre-trained Language Model for the Algerian Dialect
• Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset
• X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• Project PIAF: Building a Native French Question-Answering Dataset
• The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task
• Evaluation of contextual embeddings on less-resourced languages
• EstBERT: A Pretrained Language-Specific BERT for Estonian
• Code-mixed Sentiment and Hate-speech Prediction
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System
• GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
• Difficulty Estimation and Simplification of French Text Using LLMs
• JuriBERT: A Masked-Language Model Adaptation for French Legal Text
• UzBERT: pretraining a BERT model for Uzbek
• Can Few Lines of Code Change Society ? Beyond fack-checking and moderation : how recommender systems toxifies social networking sites
• ISA: An Intelligent Shopping Assistant
• Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan
• Towards Grad-CAM Based Explainability in a Legal Text Processing Pipeline
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• De-identification of Privacy-related Entities in Job Postings
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• On Detecting Policy-Related Political Ads: An Exploratory Analysis of Meta Ads in 2022 French Election
• Comparison of Czech Transformers on Text Classification Tasks
• More Embeddings, Better Sequence Labelers?
• AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation
• HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics
• Conversation-Based Multimodal Abuse Detection Through Text and Graph Embeddings
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Enhanced Universal Dependency Parsing with Automated Concatenation of Embeddings
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models
• Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO
• PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors
• An Incremental Clustering Baseline for Event Detection on Twitter
• BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets
• Tag and correct: high precision post-editing approach to correction of speech recognition errors
• Neural Coreference Resolution for Arabic
• TextAttack: Lessons learned in designing Python frameworks for NLP
• KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
• Where are we in semantic concept extraction for Spoken Language Understanding?
• PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords
• DPRK-BERT: The Supreme Language Model
• Thai Universal Dependency Treebank
• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
• A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition
• Towards Fully Bilingual Deep Language Modeling
• Named Entity Recognition for Monitoring Plant Health Threats in Tweets: a ChouBERT Approach
• On Inductive Biases for Machine Learning in Data Constrained Settings
• Towards a Data-Driven Requirements Engineering Approach: Automatic Analysis of User Reviews
• belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification
• Evaluating the Efficacy of Summarization Evaluation across Languages
• Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
• Structured Pattern Pruning Using Regularization
• Improved Goal Oriented Dialogue via Utterance Generation and Look Ahead