24 citations · 36 across the 15 of their papers we have counts for
15 papers
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jingwei Zuo, Cong Zeng, Ilyas Chahed +6
The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve…
Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
Maksim Velikanov, Ilyas Chahed, Jingwei Zuo +3
Applying weight decay (WD) to matrix layers is standard practice in large-language-model pretraining. Prior work suggests that stochastic gradient noise induces a Brownian-like exp…
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Jingwei Zuo, Maksim Velikanov, Ilyas Chahed +24
In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across…
Falcon Mamba: The First Competitive Attention-free 7B Language Model
Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem +4
In this technical report, we present Falcon Mamba 7B, a new base large language model based on the novel Mamba architecture. Falcon Mamba 7B is trained on 5.8 trillion tokens with…
SGD with memory: fundamental properties and stochastic acceleration
Dmitry Yarotsky, Maksim Velikanov
An important open problem is the theoretically feasible acceleration of mini-batch SGD-type algorithms on quadratic problems with power-law spectrum. In the non-stochastic setting,…
Falcon2-11B Technical Report
Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru +14
We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our f…
Generalization error of spectral algorithms
Maksim Velikanov, Maxim Panov, Dmitry Yarotsky
The asymptotically precise estimation of the generalization of kernel methods has recently received attention due to the parallels between neural networks and their associated kern…
Efficient Conformal Prediction under Data Heterogeneity
Vincent Plassier, Nikita Kotelevskii, Aleksandr Rubashevskii +7
Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification, which is crucial for ensuring the reliability of predictions. However, common CP methods…
Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks
Anastasia Antsiferova, Khaled Abud, Aleksandr Gushchin +3
Nowadays, neural-network-based image- and video-quality metrics perform better than traditional methods. However, they also became more vulnerable to adversarial attacks that incre…
A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta
Maksim Velikanov, Denis Kuznedelev, Dmitry Yarotsky
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models. In this paper we develop a new analytic framework to analyze noise-averaged properties…
Embedded Ensembles: Infinite Width Limit and Operating Regimes
Maksim Velikanov, Roman Kail, Ivan Anokhin +4
A memory efficient approach to ensembling neural networks is to share most weights among the ensembled models by means of a single reference network. We refer to this strategy as E…
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions
Maksim Velikanov, Dmitry Yarotsky
Performance of optimization on quadratic problems sensitively depends on the low-lying part of the spectrum. For large (effectively infinite-dimensional) problems, this part of the…
Universal scaling laws in the gradient descent training of neural networks
Maksim Velikanov, Dmitry Yarotsky
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss…
Proton fraction in neutron star matter: Dynamical mean-field approach
Maxim Velikanov, Alexey N. Rubtsov, Boris Krippa
Dynamical mean field theory (DMFT) is used to study neutron matter, both with and without admixture of the proton fraction. The system is approximated by the lattice Habbard model.…
Three-particle electron-hole complexes in two-dimensional electron systems
V. A. Kuznetsov, L. V. Kulik, M. D. Velikanov +4
Three-particle complexes consisting of two holes in the completely filled zero electron Landau level and an excited electron in the unoccupied first Landau level are investigated i…