4 papers
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
Anand Gopalakrishnan, Robert Csordás, Jürgen Schmidhuber +1
The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indi…
Mindstorms in Natural Language-Based Societies of Mind
Mingchen Zhuge, Haozhe Liu, Francesco Faccio +23
Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each o…
Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery
Anand Gopalakrishnan, Aleksandar StaniÄ, Jürgen Schmidhuber +1
Current state-of-the-art synchrony-based models encode object bindings with complex-valued activations and compute with real-valued weights in feedforward architectures. We argue f…
Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning
Etai Littwin, Vimal Thilak, Anand Gopalakrishnan
Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) for representation learning using the Masked Image Modeling…