61 citations · 150 across the 17 of their papers we have counts for
17 papers
Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignme…
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contri…
GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
Anand Choudhary, Yasser Sulaıman, Lukas Mauch +3
Sparse fine-tuning techniques adapt LLMs to downstream tasks by only tuning a sparse subset of model parameters. However, the effectiveness of sparse adaptation depends on optimall…
SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning
Bac Nguyen, Stefan Uhlich, Fabien Cardinaux +3
Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trai…
LLM meets Vision-Language Models for Zero-Shot One-Class Classification
Yassir Bendou, Giulia Lioi, Bastien Pasdeloup +4
We consider the problem of zero-shot one-class visual classification, extending traditional one-class classification to scenarios where only the label of the target class is availa…
A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models
Reda Bensaid, Vincent Gripon, François Leduc-Primeau +3
Few-shot semantic segmentation (FSS) is a crucial challenge in computer vision, driving extensive research into a diverse range of methods, from advanced meta-learning techniques t…
Inferring Latent Class Statistics from Text for Robust Visual Few-Shot Learning
Yassir Bendou, Vincent Gripon, Bastien Pasdeloup +4
In the realm of few-shot learning, foundation models like CLIP have proven effective but exhibit limitations in cross-domain robustness especially in few-shot settings. Recent work…
DBsurf: A Discrepancy Based Method for Discrete Stochastic Gradient Estimation
Pau Mulet Arabi, Alec Flowers, Lukas Mauch +1
Computing gradients of an expectation with respect to the distributional parameters of a discrete distribution is a problem arising in many fields of science and engineering. Typic…
Efficient Training of Deep Equilibrium Models
Bac Nguyen, Lukas Mauch
Deep equilibrium models (DEQs) have proven to be very powerful for learning data representations. The idea is to replace traditional (explicit) feedforward neural networks with an…
A Statistical Model for Predicting Generalization in Few-Shot Classification
Yassir Bendou, Vincent Gripon, Bastien Pasdeloup +5
The estimation of the generalization error of classifiers often relies on a validation set. Such a set is hardly available in few-shot learning scenarios, a highly disregarded shor…
DNN Quantization with Attention
Ghouthi Boukli Hacene, Lukas Mauch, Stefan Uhlich +1
Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). Howe…
Efficient Sampling for Predictor-Based Neural Architecture Search
Lukas Mauch, Stephen Tiedemann, Javier Alonso Garcia +4
Recently, predictor-based algorithms emerged as a promising approach for neural architecture search (NAS). For NAS, we typically have to calculate the validation accuracy of a larg…
Iteratively Training Look-Up Tables for Network Quantization
Fabien Cardinaux, Stefan Uhlich, Kazuki Yoshiyama +5
Operating deep neural networks (DNNs) on devices with limited resources requires the reduction of their memory as well as computational footprint. Popular reduction methods are net…
Mixed Precision DNNs: All you need is a good parametrization
Stefan Uhlich, Lukas Mauch, Fabien Cardinaux +5
Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precisio…
Deep Neural Network inference with reduced word length
Lukas Mauch, Bin Yang
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high…
A Machine-learning framework for automatic reference-free quality assessment in MRI
Thomas Küstner, Sergios Gatidis, Annika Liebgott +9
Magnetic resonance (MR) imaging offers a wide variety of imaging techniques. A large amount of data is created per examination which needs to be checked for sufficient quality in o…
Neural Network Ensembles to Real-time Identification of Plug-level Appliance Measurements
Karim Said Barsim, Lukas Mauch, Bin Yang
The problem of identifying end-use electrical appliances from their individual consumption profiles, known as the appliance identification problem, is a primary stage in both Non-I…