157 citations · 288 across the 7 of their papers we have counts for
10 papers
Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment
Ziao Wang, Kilian Müller, Matthew Filipovich +12
Modern deep learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation speed, are increasingly c…
The Falcon Series of Open Language Models
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi +11
We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest…
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Guilherme Penedo, Quentin Malartic, Daniel Hesslow +6
Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This…
Scaling Laws Beyond Backpropagation
Matthew J. Filipovich, Alessandro Cappelli, Daniel Hesslow +1
Alternatives to backpropagation have long been studied to better understand how biological brains may learn. Recently, they have also garnered interest as a way to train neural net…
PAGnol: An Extra-Large French Generative Model
Julien Launay, Elena Tommasone, Baptiste Pannier +5
Access to large pre-trained models of varied architectures, in many different languages, is central to the democratization of NLP. We introduce PAGnol, a collection of French GPT m…
LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor
Charles Brossollet, Alessandro Cappelli, Igor Carron +14
We introduce LightOn's Optical Processing Unit (OPU), the first photonic AI accelerator chip available on the market for at-scale Non von Neumann computations, reaching 1500 TeraOP…
ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients
Alessandro Cappelli, Julien Launay, Laurent Meunier +2
Robustness to adversarial attacks is typically obtained through expensive adversarial training with Projected Gradient Descent. Here we introduce ROPUST, a remarkably simple and ef…
Photonic Differential Privacy with Direct Feedback Alignment
Ruben Ohana, Hamlet J. Medina Ruiz, Julien Launay +4
Optical Processing Units (OPUs) -- low-power photonic chips dedicated to large scale random projections -- have been used in previous work to train deep neural networks using Direc…
Photonic co-processors in HPC: using LightOn OPUs for Randomized Numerical Linear Algebra
Daniel Hesslow, Alessandro Cappelli, Igor Carron +6
Randomized Numerical Linear Algebra (RandNLA) is a powerful class of methods, widely used in High Performance Computing (HPC). RandNLA provides approximate solutions to linear alge…
Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
Alessandro Cappelli, Ruben Ohana, Julien Launay +3
We propose a new defense mechanism against adversarial attacks inspired by an optical co-processor, providing robustness without compromising natural accuracy in both white-box and…