6 papers
Stacked LoRA for Subject-Adaptive EEG Foundation Models in Motor Imagery Decoding
Aymen Sarhane, Fouad Lbakali, Mouad Souissi +2
Electroencephalography (EEG) decoding for brain-computer interfaces (BCIs) faces a major challenge: substantial inter-subject variability limits effective cross-subject generalizat…
D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding
Jonathan Lys, Vincent Gripon, Axel Marmoret +4
Discrete diffusion models are promising alternatives to autoregressive approaches for text generation, yet their decoding methods remain under-studied. Standard autoregressive sear…
Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignme…
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4
Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contri…
TensLoRA: Tensor Alternatives for Low-Rank Adaptation
Axel Marmoret, Reda Bensaid, Jonathan Lys +2
Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are cons…
REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects
Yassine El Ouahidi, Jonathan Lys, Philipp Thölke +5
Foundation models have transformed AI by reducing reliance on task-specific data through large-scale pretraining. While successful in language and vision, their adoption in EEG has…