104 citations · 673 across the 47 of their papers we have counts for
22 papers · 1 filter
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Maria Lomeli, Antoine Groudiev, Matthijs Douze +7
This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference…
Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks
Mariia Drozdova, Stéphane Liem Nguyen, François Fleuret
Denoising Diffusion Probabilistic Models (DDPMs) generate samples by starting from noise and repeatedly denoising while keeping each update close to the current noisy state. This b…
End-to-End Optimal Detector Design with Mutual Information Surrogates
Kinga Anna Wozniak, Stephen Mulligan, Jan Kieseler +3
We introduce a novel approach for end-to-end black-box optimization of high energy physics (HEP) detectors using local deep learning (DL) surrogates. These surrogates approximate a…
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
Ke Wang, Nikolaos Dimitriadis, Alessandro Favero +3
Fine-tuning pre-trained models has become the standard approach to endow them with specialized knowledge, but it poses fundamental challenges. In particular, \textit{(i)} fine-tuni…
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences
Nikolaos Dimitriadis, Pascal Frossard, Francois Fleuret
Multi-task trade-offs in machine learning can be addressed via Pareto Front Learning (PFL) methods that parameterize the Pareto Front (PF) with a single model. PFL permits to selec…
DeepEMD: A Transformer-based Fast Estimation of the Earth Mover's Distance
Atul Kumar Sinha, Francois Fleuret
The Earth Mover's Distance (EMD) is the measure of choice between point clouds. However the computational cost to compute it makes it prohibitive as a training loss, and the standa…
SequeL: A Continual Learning Library in PyTorch and JAX
Nikolaos Dimitriadis, Francois Fleuret, Pascal Frossard
Continual Learning is an important and challenging problem in machine learning, where models must adapt to a continuous stream of new data without forgetting previously acquired kn…
Inference from Real-World Sparse Measurements
Arnaud Pannatier, Kyle Matoba, François Fleuret
Real-world problems often involve complex and unstructured sets of measurements, which occur when sensors are sparsely placed in either space or time. Being able to model this irre…
Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models
Nikolaos Dimitriadis, Pascal Frossard, François Fleuret
In Multi-Task Learning (MTL), tasks may compete and limit the performance achieved on each other, rather than guiding the optimization to a solution, superior to all its single-tas…
Transformers are Sample-Efficient World Models
Vincent Micheli, Eloi Alonso, François Fleuret
Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have b…
Efficiently Training Low-Curvature Neural Networks
Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju +1
The highly non-linear nature of deep neural networks causes them to be susceptible to adversarial examples and have unstable gradients which hinders interpretability. However, exis…
Flowification: Everything is a Normalizing Flow
Bálint Máté, Samuel Klein, Tobias Golling +1
The two key characteristics of a normalizing flow is that it is invertible (in particular, dimension preserving) and that it monitors the amount by which it changes the likelihood…
Agree to Disagree: Diversity through Disagreement for Better Transferability
Matteo Pagliardini, Martin Jaggi, François Fleuret +1
Gradient-based learning algorithms have an implicit simplicity bias which in effect can limit the diversity of predictors being sampled by the learning procedure. This behavior can…
Efficient Wind Speed Nowcasting with GPU-Accelerated Nearest Neighbors Algorithm
Arnaud Pannatier, Ricardo Picatoste, François Fleuret
This paper proposes a simple yet efficient high-altitude wind nowcasting pipeline. It processes efficiently a vast amount of live data recorded by airplanes over the whole airspace…
Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
Suraj Srinivas, Francois Fleuret
Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t. the in…
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels +2
The performance of optimizers, particularly in deep learning, depends considerably on their chosen hyperparameter configuration. The efficacy of optimizers is often studied under n…
Full-Gradient Representation for Neural Network Visualization
Suraj Srinivas, Francois Fleuret
We introduce a new tool for interpreting neural net responses, namely full-gradients, which decomposes the neural net response into input sensitivity and per-neuron sensitivity com…
Not All Samples Are Created Equal: Deep Learning with Importance Sampling
Angelos Katharopoulos, François Fleuret
Deep neural network training spends most of the computation on examples that are properly handled, and could be ignored. We propose to mitigate this phenomenon with a principled im…
Knowledge Transfer with Jacobian Matching
Suraj Srinivas, Francois Fleuret
Classical distillation methods transfer representations from a "teacher" neural network to a "student" network by matching their output activations. Recent methods also match the J…
Biased Importance Sampling for Deep Neural Network Training
Angelos Katharopoulos, François Fleuret
Importance sampling has been successfully used to accelerate stochastic optimization in many convex problems. However, the lack of an efficient way to calculate the importance stil…
Kronecker Recurrent Units
Cijo Jose, Moustpaha Cisse, Francois Fleuret
Our work addresses two important issues with recurrent neural networks: (1) they are over-parameterized, and (2) the recurrence matrix is ill-conditioned. The former increases the…
A provably convergent alternating minimization method for mean field inference
Pierre Baqué, Jean-Hubert Hours, François Fleuret +1
Mean-Field is an efficient way to approximate a posterior distribution in complex graphical models and constitutes the most popular class of Bayesian variational approximation meth…