72 citations · 289 across the 69 of their papers we have counts for
9 papers · 1 filter
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
Shachar Don-Yehiya, Elad Venezian, Colin Raffel +3
We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion. It provides the benefits of multitask learning but leverages distributed computation with li…
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Ella Neeman, Roee Aharoni, Or Honovich +3
Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledge - the factual knowledge encoded in the model weights, a…
Where to start? Analyzing the potential value of intermediate models
Leshem Choshen, Elad Venezian, Shachar Don-Yehia +2
Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better st…
Reinforcement Learning with Large Action Spaces for Neural Machine Translation
Asaf Yehudai, Leshem Choshen, Lior Fox +1
Applying Reinforcement learning (RL) following maximum likelihood estimation (MLE) pre-training is a versatile method for enhancing neural machine translation (NMT) performance. Ho…
Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours
Eyal Shnarch, Alon Halfon, Ariel Gera +19
Text classification can be useful in many real-world scenarios, saving a lot of time for end users. However, building a custom classifier typically requires coding skills and ML kn…
Some Grammatical Errors are Frequent, Others are Important
Leshem Choshen, Ofir Shifman, Omri Abend
In Grammatical Error Correction, systems are evaluated by the number of errors they correct. However, no one has assessed whether all error types are equally important. We provide…
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance
Shachar Don-Yehiya, Leshem Choshen, Omri Abend
We present the task of PreQuEL, Pre-(Quality-Estimation) Learning. A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translatio…
Fusing finetuned models for better pretraining
Leshem Choshen, Elad Venezian, Noam Slonim +1
Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavou…
Cluster & Tune: Boost Cold Start Performance in Text Classification
Eyal Shnarch, Ariel Gera, Alon Halfon +4
In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce. In such cases, the common practice of fine-tuning pre-trained model…