4 papers
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Sajad Movahedi, Vera MilovanoviÄ, Shlomo Libo Feigin +5
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by loo…
Latent-DARM: Bridging Discrete Diffusion And Autoregressive Models For Reasoning
Lina Berrayana, Ahmed Heakl, Abdullah Sohail +3
Most multi-agent systems rely exclusively on autoregressive language models (ARMs) that are based on sequential generation. Although effective for fluent text, ARMs limit global re…
Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models
Aleksandar TerziÄ, Nicolas Menet, Michael Hersche +2
Modern state-space models (SSMs) often utilize transition matrices which enable efficient computation but pose restrictions on the model's expressivity, as measured in terms of the…
On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
Aleksandar TerziÄ, Michael Hersche, Giacomo Camposampiero +3
Selective state-space models (SSMs) are an emerging alternative to the Transformer, offering the unique advantage of parallel training and sequential inference. Although these mode…