8 papers
WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention
Ruben Solozabal, Velibor Bojkovic, Hilal Alquabeh +3
State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be…
Uncovering the Spectral Bias in Diagonal State Space Models
Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh +2
Current methods for initializing state space models (SSMs) parameters mainly rely on the \textit{HiPPO framework}, which is based on an online approximation of orthogonal polynomia…
SPACT18: Spiking Human Action Recognition Benchmark Dataset with Complementary RGB and Thermal Modalities
Yasser Ashraf, Ahmed Sharshar, Velibor Bojkovic +1
Spike cameras, bio-inspired vision sensors, asynchronously fire spikes by accumulating light intensities at each pixel, offering ultra-high energy efficiency and exceptional tempor…
Mechanistic Insights into Grokking from the Embedding Layer
H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic +2
Grokking, a delayed generalization in neural networks after perfect training performance, has been observed in Transformers and MLPs, but the components driving it remain underexpl…
Number Representations in LLMs: A Computational Parallel to Human Perception
H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic +4
Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, sup…
RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu +6
We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirection…