5 citations · 5 across the 8 of their papers we have counts for
4 papers · 1 filter
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Jarod Lévy, Mathurin Videau, Jad Yehya +3
Transformers process tokens without any inherent notion of order, making positional encoding a fundamental requirement rather than an architectural refinement. Rotary Position Embe…
Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes.…
Separating Representation from Reconstruction Enables Scalable Text Encoders
Megi Dervishi, Mathurin Videau, Yann LeCun
While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the r…
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
João Maria Janeiro, Mathurin Videau, Andrea Caciolai +3
Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically…