8 papers
On the Stability of the Jacobian Matrix in Deep Neural Networks
Benjamin Dadoun, Soufiane Hayou, Hanan Salam +2
Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian.…
Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik +1
Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitiv…
High-Dimensional Analysis of Bootstrap Ensemble Classifiers
Malik Tiomoko, Hamza Cherkaoui, Mohamed El Amine Seddik +3
Bootstrap methods have long been the cornerstone of ensemble learning in machine learning. This paper presents a theoretical analysis of bootstrap techniques applied to the Least S…
How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models
Mohamed El Amine Seddik
We study the spectral properties of sample covariance matrices constructed from pooled sequence representations, where token embeddings are drawn from a fixed two-class Gaussian mi…
Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling
Falcon LLM Team, Iheb Chaabane, Puneesh Khanna +8
This work introduces Falcon-H1R, a 7B-parameter reasoning-optimized model that establishes the feasibility of achieving competitive reasoning performance with small language models…
-LoRA: Effective Fine-Tuning via Base Model Rescaling
Aymane El Firdoussi, El Mahdi Chayti, Mohamed El Amine Seddik +1
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches…