5 papers
Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes
Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó
An overcomplete code packs features into dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor $W(F,d)=(…
Component-Aware Self-Speculative Decoding in Hybrid Language Models
Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó
Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods av…
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó
Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniforml…
How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models
Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó
Weight pruning is a standard technique for compressing large language models, yet its effect on learned internal representations remains poorly understood. We present the first sys…
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó
Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each compon…