6 citations · 6 across the 4 of their papers we have counts for
3 papers · 1 filter
Structural analysis of an all-purpose question answering model
Vincent Micheli, Quentin Heinrich, François Fleuret +1
Attention is a key component of the now ubiquitous pre-trained language models. By learning to focus on relevant pieces of information, these Transformer-based architectures have p…
Language Models are Few-Shot Butlers
Vincent Micheli, François Fleuret
Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. Hence, these autoregressive models constitute ideal age…
On the importance of pre-training data volume for compact language models
Vincent Micheli, Martin d'Hoffschmidt, François Fleuret
Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the…