59 citations · 152 across the 14 of their papers we have counts for
18 papers
Large-Scale AI in Telecom: Charting the Roadmap for Innovation, Scalability, and Enhanced Digital Experiences
Adnan Shahid, Adrian Kliks, Ahmed Al-Tahmeesschi +132
This white paper discusses the role of large-scale AI in the telecommunications industry, with a specific focus on the potential of generative AI to revolutionize network functions…
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Benjamin Warner, Antoine Chaffin, Benjamin Clavié +11
Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite bein…
Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment
Ziao Wang, Kilian Müller, Matthew Filipovich +12
Modern deep learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation speed, are increasingly c…
RITA: a Study on Scaling Up Generative Protein Sequence Models
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin +2
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences…
PAGnol: An Extra-Large French Generative Model
Julien Launay, Elena Tommasone, Baptiste Pannier +5
Access to large pre-trained models of varied architectures, in many different languages, is central to the democratization of NLP. We introduce PAGnol, a collection of French GPT m…
Is the Number of Trainable Parameters All That Actually Matters?
Amélie Chatelain, Amine Djeghri, Daniel Hesslow +2
Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss. The validity of th…
LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor
Charles Brossollet, Alessandro Cappelli, Igor Carron +14
We introduce LightOn's Optical Processing Unit (OPU), the first photonic AI accelerator chip available on the market for at-scale Non von Neumann computations, reaching 1500 TeraOP…
ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients
Alessandro Cappelli, Julien Launay, Laurent Meunier +2
Robustness to adversarial attacks is typically obtained through expensive adversarial training with Projected Gradient Descent. Here we introduce ROPUST, a remarkably simple and ef…
Photonic Differential Privacy with Direct Feedback Alignment
Ruben Ohana, Hamlet J. Medina Ruiz, Julien Launay +4
Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to train deep neural networks using Direc…
Photonic co-processors in HPC: using LightOn OPUs for Randomized Numerical Linear Algebra
Daniel Hesslow, Alessandro Cappelli, Igor Carron +6
Randomized Numerical Linear Algebra (RandNLA) is a powerful class of methods, widely used in High Performance Computing (HPC). RandNLA provides approximate solutions to linear alge…
Contrastive Embeddings for Neural Architectures
Daniel Hesslow, Iacopo Poli
The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across di…
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
Alessandro Cappelli, Ruben Ohana, Julien Launay +3
We propose a new defense mechanism against adversarial attacks inspired by an optical co-processor, providing robustness without compromising natural accuracy in both white-box and…
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
Julien Launay, Iacopo Poli, Kilian Müller +5
The scaling hypothesis motivates the expansion of models past trillions of parameters as a path towards better performance. Recent significant developments, such as GPT-3, have bee…
Online Change Point Detection in Molecular Dynamics With Optical Random Features
Amélie Chatelain, Elena Tommasone, Laurent Daudet +1
Proteins are made of atoms constantly fluctuating, but can occasionally undergo large-scale changes. Such transitions are of biological interest, linking the structure of a protein…
Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
Julien Launay, Iacopo Poli, Kilian Müller +4
As neural networks grow larger and more complex and data-hungry, training costs are skyrocketing. Especially when lifelong learning is necessary, such as in recommender systems or…
Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
Julien Launay, Iacopo Poli, François Boniface +1
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea. It enforces sequential layer updates, thus preventing efficient parallelization of the tr…
Principled Training of Neural Networks with Direct Feedback Alignment
Julien Launay, Iacopo Poli, Florent Krzakala
The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to en…
NEWMA: a new method for scalable model-free online change-point detection
Nicolas Keriven, Damien Garreau, Iacopo Poli
We consider the problem of detecting abrupt changes in the distribution of a multi-dimensional time series, with limited computing power and memory. In this paper, we propose a new…