46 citations · 101 across the 25 of their papers we have counts for
25 papers
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
Anand Gopalakrishnan, Robert Csordás, Jürgen Schmidhuber +1
The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indi…
Directly Forecasting Belief for Reinforcement Learning with Delays
Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan +6
Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based o…
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
Piotr Piękos, Róbert Csordás, Jürgen Schmidhuber
Recent advances in large language models highlighted the excessive quadratic cost of self-attention. Despite the significant research efforts, subquadratic attention methods still…
Measuring In-Context Computation Complexity via Hidden State Prediction
Vincent Herrmann, Róbert Csordás, Jürgen Schmidhuber
Detecting when a neural sequence model does "interesting" computation is an open problem. The next token prediction loss is a poor indicator: Low loss can stem from trivially predi…
Automatic Album Sequencing
Vincent Herrmann, Dylan R. Ashley, Jürgen Schmidhuber
Album sequencing is a critical part of the album production process. Recently, a data-driven approach was proposed that sequences general collections of independent media by extrac…
Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery
Anand Gopalakrishnan, Aleksandar Stanić, Jürgen Schmidhuber +1
Current state-of-the-art synchrony-based models encode object bindings with complex-valued activations and compute with real-valued weights in feedforward architectures. We argue f…