52 citations · 69 across the 6 of their papers we have counts for
9 papers · 1 filter
Boolformer: Symbolic Regression of Logic Functions with Transformers
Stéphane d'Ascoli, Arthur Renard, Vassilis Papadopoulos +3
We introduce Boolformer, a Transformer-based model trained to perform end-to-end symbolic regression of Boolean functions. First, we show that it can predict compact formulas for c…
Length Generalization in Arithmetic Transformers
Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich +3
We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative posit…
End-to-end symbolic regression with transformers
Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample +1
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step proc…
Optimal learning rate schedules in high-dimensional non-convex optimization problems
Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli
Learning rate schedules are ubiquitously used to speed up and improve optimisation. Many different policies have been introduced on an empirical basis, and theoretical analyses hav…
Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Stéphane d'Ascoli, Levent Sagun, Giulio Biroli +1
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt to bridge the gap between these two archi…
On the interplay between data structure and loss function in classification problems
Stéphane d'Ascoli, Marylou Gabrié, Levent Sagun +1
One of the central puzzles in modern machine learning is the ability of heavily overparametrized models to generalize well. Although the low-dimensional structure of typical datase…