5 citations · 5 across the 8 of their papers we have counts for
11 papers
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Jarod Lévy, Mathurin Videau, Jad Yehya +3
Transformers process tokens without any inherent notion of order, making positional encoding a fundamental requirement rather than an architectural refinement. Rotary Position Embe…
Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes.…
Separating Representation from Reconstruction Enables Scalable Text Encoders
Megi Dervishi, Mathurin Videau, Yann LeCun
While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the r…
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
João Maria Janeiro, Mathurin Videau, Andrea Caciolai +3
Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically…
Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
Ismail Labiad, Mathurin Videau, Matthieu Kowalski +4
Gradient-based optimization is the workhorse of deep learning, offering efficient and scalable training via backpropagation. However, exposing gradients during training can leak se…
Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
Wassim Bouaziz, Mathurin Videau, Nicolas Usunier +1
The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hid…
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite +3
Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and simil…
Evolutionary Pre-Prompt Optimization for Mathematical Reasoning
Mathurin Videau, Alessandro Leite, Marc Schoenauer +1
Recent advancements have highlighted that large language models (LLMs), when given a small set of task-specific examples, demonstrate remarkable proficiency, a capability that exte…
Mixture of Experts in Image Classification: What's the Sweet Spot?
Mathurin Videau, Alessandro Leite, Marc Schoenauer +1
Mixture-of-Experts (MoE) models have shown promising potential for parameter-efficient scaling across domains. However, their application to image classification remains limited, o…
Evolutionary Retrofitting
Mathurin Videau, Mariia Zameshina, Alessandro Leite +3
AfterLearnER (After Learning Evolutionary Retrofitting) consists in applying evolutionary optimization to refine fully trained machine learning models by optimizing a set of carefu…
Improving Nevergrad's Algorithm Selection Wizard NGOpt through Automated Algorithm Configuration
Risto Trajanov, Ana Nikolikj, Gjorgjina Cenikj +6
Algorithm selection wizards are effective and versatile tools that automatically select an optimization algorithm given high-level information about the problem and available compu…