26 citations · 44 across the 8 of their papers we have counts for
9 papers
Transformers know more than they can tell -- Learning the Collatz sequence
François Charton, Ashvni Narayanan
We investigate transformer prediction of long Collatz steps, a complex arithmetic function that maps odd integers to their distant successors in the Collatz sequence ( $u_{n+1}=u_n…
Learning Euler Factors of Elliptic Curves
Angelica Babei, François Charton, Edgar Costa +5
We apply transformer models and feedforward neural networks to predict Frobenius traces from elliptic curves given other traces . We train further models to predict $a_p…
Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmetic
Eshika Saxena, Alberto Alfarano, François Charton +3
Recent work showed that ML-based attacks on Learning with Errors (LWE), a hard problem used in post-quantum cryptography, outperform classical algebraic attacks in certain settings…
Length Generalization in Arithmetic Transformers
Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich +3
We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative posit…
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning
Chris Lengerich, Gabriel Synnaeve, Amy Zhang +4
Traditional approaches to RL have focused on learning decision policies directly from episodic decisions, while slowly and implicitly learning the semantics of compositional repres…
What is my math transformer doing? -- Three results on interpretability and generalization
François Charton
This paper investigates the failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition. I show that incorrect model pre…
End-to-end symbolic regression with transformers
Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample +1
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step proc…
Deep Symbolic Regression for Recurrent Sequences
Stéphane d'Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample +1
Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the functi…
Learning advanced mathematical computations from examples
François Charton, Amaury Hayat, Guillaume Lample
Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and contro…