5 papers
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Hector Borobia, Elies SeguÃ-Mas, Guillermina Tormo-Carbó
Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each compon…
Linear-Readout Floors and Threshold Recovery in Computation in Superposition
Hector Borobia, Elies SeguÃ-Mas, Guillermina Tormo-Carbó
Two recent approaches to computation in superposition reach different recursive capacity regimes: Hänni et al. certify computable features in width via an…
Component-Aware Self-Speculative Decoding in Hybrid Language Models
Hector Borobia, Elies SeguÃ-Mas, Guillermina Tormo-Carbó
Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods av…
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Hector Borobia, Elies SeguÃ-Mas, Guillermina Tormo-Carbó
Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniforml…
How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models
Hector Borobia, Elies SeguÃ-Mas, Guillermina Tormo-Carbó
Weight pruning is a standard technique for compressing large language models, yet its effect on learned internal representations remains poorly understood. We present the first sys…