10 papers
D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding
Jonathan Lys, Vincent Gripon, Axel Marmoret +4
Discrete diffusion models are promising alternatives to autoregressive approaches for text generation, yet their decoding methods remain under-studied. Standard autoregressive sear…
SLAD : Shared LoRA Adapters for Task Specific Distillation
Reda Bensaid, Yassir Bendou, Vincent Gripon +1
In the context of resource-constrained environments such as embedded systems, adapting reduced-size foundation models to downstream tasks has become increasingly popular. This has…
Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT
Shuhei Tsuyuki, Reda Bensaid, Jérémy Morlier +4
Efficient and adaptable deep learning models are an important area of deep learning research, driven by the need for highly efficient models on edge devices. Few-shot learning enab…
Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignme…
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contri…
MUKA: Multi Kernel Audio Adaptation Of Audio-Language Models
Reda Bensaid, Amine Ouasfi, Yassir Bendou +4
Multimodal foundation models have demonstrated impressive generalization capabilities, yet efficiently adapting them to new tasks in a few-shot setting remains a critical challenge…