fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Abstract

fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto

NAACL 2019 Demo paper

References in corpus (13)

• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Sockeye: A Toolkit for Neural Machine Translation
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Weighted Transformer Network for Machine Translation
• An Analysis of Neural Language Modeling at Multiple Scales
• Bottom-Up Abstractive Summarization
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
• Fast Parametric Learning with Activation Memorization
• Double Path Networks for Sequence to Sequence Learning
• A Stable and Effective Learning Strategy for Trainable Greedy Decoding

Cited by in corpus (104)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• BERTScore: Evaluating Text Generation with BERT
• Longformer: The Long-Document Transformer
• fastai: A Layered API for Deep Learning
• Multilingual Denoising Pre-training for Neural Machine Translation
• On the Variance of the Adaptive Learning Rate and Beyond
• SpeechBrain: A General-Purpose Speech Toolkit
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• Recipes for Safety in Open-domain Chatbots
• Very Deep Transformers for Neural Machine Translation
• Rethinking Positional Encoding in Language Pre-training
• Hash Layers For Large Sparse Models
• Generating Bug-Fixes Using Pretrained Transformers
• Self-training Improves Pre-training for Natural Language Understanding
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• When Does Unsupervised Machine Translation Work?
• Addressing Some Limitations of Transformers with Feedback Memory
• A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
• JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus
• Nearest Neighbor Machine Translation
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• AmbigQA: Answering Ambiguous Open-domain Questions
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Blockwise Self-Attention for Long Document Understanding
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Multi-Stage Pre-training for Low-Resource Domain Adaptation
• Low-Resource Task-Oriented Semantic Parsing via Intrinsic Modeling
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Cascaded Text Generation with Markov Transformers
• An Efficient Transformer Decoder with Compressed Sub-layers
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
• LazyFormer: Self Attention with Lazy Update
• Improving Conditioning in Context-Aware Sequence to Sequence Models
• Editable Neural Networks
• Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
• Learning Shared Semantic Space for Speech-to-Text Translation
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• A Comparison of Approaches to Document-level Machine Translation
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
• Contextual Text Denoising with Masked Language Models
• Content Planning for Neural Story Generation with Aristotelian Rescoring
• Optimizing Prediction Serving on Low-Latency Serverless Dataflow
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• Sub-Linear Memory: How to Make Performers SLiM
• Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Multitask Learning for Class-Imbalanced Discourse Classification
• Neural Language Priors
• Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation
• MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
• Spanish Legalese Language Model and Corpora
• FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation
• Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
• 8-bit Optimizers via Block-wise Quantization
• Taking Notes on the Fly Helps BERT Pre-training
• Self-Supervised Representations Improve End-to-End Speech Translation
• Probing Acoustic Representations for Phonetic Properties
• Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs
• AMR Parsing with Action-Pointer Transformer
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Reduce and Reconstruct: ASR for Low-Resource Phonetic Languages
• Computational Morphology with Neural Network Approaches
• Unsupervised Text Summarization via Mixed Model Back-Translation
• What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding
• Meta-Learning for Few-Shot NMT Adaptation
• Future-Guided Incremental Transformer for Simultaneous Translation
• Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks
• I-BERT: Inductive Generalization of Transformer to Arbitrary Context Lengths
• Controllable Open-ended Question Generation with A New Question Type Ontology
• Revisiting Modularized Multilingual NMT to Meet Industrial Demands
• Search Spaces for Neural Model Training
• Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding
• Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
• The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
• Pattern-aware Data Augmentation for Query Rewriting in Voice Assistant Systems
• Interpreting A Pre-trained Model Is A Key For Model Architecture Optimization: A Case Study On Wav2Vec 2.0
• STC speaker recognition systems for the NIST SRE 2021
• Autoregressive Knowledge Distillation through Imitation Learning
• Reciprocal Supervised Learning Improves Neural Machine Translation
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• A Dynamic Head Importance Computation Mechanism for Neural Machine Translation
• On Anytime Learning at Macroscale
• Sketch and Refine: Towards Faithful and Informative Table-to-Text Generation
• Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation
• Learn to Talk via Proactive Knowledge Transfer
• Improving Multilingual Translation by Representation and Gradient Regularization
• Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation
• Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation
• Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing