157 citations · 371 across the 9 of their papers we have counts for
12 papers
The Falcon Series of Open Language Models
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi +11
We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest…
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Guilherme Penedo, Quentin Malartic, Daniel Hesslow +6
Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This…
Linear Optical Random Projections Without Holography
Ruben Ohana, Daniel Hesslow, Daniel Brunner +2
We introduce a novel method to perform linear optical random projections without the need for holography. Our method consists of a computationally trivial combination of multiple i…
What Language Model to Train if You Have One Million GPU Hours?
Teven Le Scao, Thomas Wang, Daniel Hesslow +16
The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across t…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
Scaling Laws Beyond Backpropagation
Matthew J. Filipovich, Alessandro Cappelli, Daniel Hesslow +1
Alternatives to backpropagation have long been studied to better understand how biological brains may learn. Recently, they have also garnered interest as a way to train neural net…
RITA: a Study on Scaling Up Generative Protein Sequence Models
Daniel Hesslow, Niccoló Zanichelli, Pascal Notin +2
In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences…
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
Thomas Wang, Adam Roberts, Daniel Hesslow +5
Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly traine…
Is the Number of Trainable Parameters All That Actually Matters?
Amélie Chatelain, Amine Djeghri, Daniel Hesslow +2
Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss. The validity of th…
LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor
Charles Brossollet, Alessandro Cappelli, Igor Carron +14
We introduce LightOn's Optical Processing Unit (OPU), the first photonic AI accelerator chip available on the market for at-scale Non von Neumann computations, reaching 1500 TeraOP…
Photonic co-processors in HPC: using LightOn OPUs for Randomized Numerical Linear Algebra
Daniel Hesslow, Alessandro Cappelli, Igor Carron +6
Randomized Numerical Linear Algebra (RandNLA) is a powerful class of methods, widely used in High Performance Computing (HPC). RandNLA provides approximate solutions to linear alge…
Contrastive Embeddings for Neural Architectures
Daniel Hesslow, Iacopo Poli
The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across di…