Efficient transfer learning for NLP with ELECTRA
arXiv:2104.02756
Abstract
Clark et al. [2020] claims that the ELECTRA approach is highly efficient in NLP performances relative to computation budget. As such, this reproducibility study focus on this claim, summarized by the following question: Can we use ELECTRA to achieve close to SOTA performances for NLP in low-resource settings, in term of compute cost?
arXiv commentSubmission for ML Reproducibility Challenge 2020
References in corpus (5)
- Adam: A Method for Stochastic Optimization
- Decoupled Weight Decay Regularization
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- On the comparability of Pre-trained Language Models