9 papers
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
Anand Gopalakrishnan, Robert Csordás, Jürgen Schmidhuber +1
The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indi…
Directly Forecasting Belief for Reinforcement Learning with Delays
Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan +6
Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based o…
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
Piotr PiÄkos, Róbert Csordás, Jürgen Schmidhuber
Recent advances in large language models highlighted the excessive quadratic cost of self-attention. Despite the significant research efforts, subquadratic attention methods still…
Measuring In-Context Computation Complexity via Hidden State Prediction
Vincent Herrmann, Róbert Csordás, Jürgen Schmidhuber
Detecting when a neural sequence model does "interesting" computation is an open problem. The next token prediction loss is a poor indicator: Low loss can stem from trivially predi…
Metalearning Continual Learning Algorithms
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments. Conventional learning algorithms for neural networks, however, suffe…
Automatic Album Sequencing
Vincent Herrmann, Dylan R. Ashley, Jürgen Schmidhuber
Album sequencing is a critical part of the album production process. Recently, a data-driven approach was proposed that sequences general collections of independent media by extrac…