743 citations · 1.4k across the 18 of their papers we have counts for
18 papers
Boolformer: Symbolic Regression of Logic Functions with Transformers
Stéphane d'Ascoli, Arthur Renard, Vassilis Papadopoulos +3
We introduce Boolformer, a Transformer-based model trained to perform end-to-end symbolic regression of Boolean functions. First, we show that it can predict compact formulas for c…
Length Generalization in Arithmetic Transformers
Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich +3
We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative posit…
End-to-end symbolic regression with transformers
Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample +1
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step proc…
Optimal learning rate schedules in high-dimensional non-convex optimization problems
Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli
Learning rate schedules are ubiquitously used to speed up and improve optimisation. Many different policies have been introduced on an empirical basis, and theoretical analyses hav…
Deep Symbolic Regression for Recurrent Sequences
Stéphane d'Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample +1
Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the functi…
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Stéphane d'Ascoli, Levent Sagun, Giulio Biroli +1
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt to bridge the gap between these two archi…