54 citations · 79 across the 11 of their papers we have counts for
5 papers · 1 filter
Phases in a class of associative memories via hidden neurons
Toshihiro Ota, Masato Taki
Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softma…
On the Role of Hidden States of Modern Hopfield Network in Transformer
Tsubasa Masumura, Masato Taki
Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learnin…
Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking
Ryo Karakida, Toshihiro Ota, Masato Taki
Transformers have established themselves as the leading neural network model in natural language processing and are increasingly foundational in various domains. In vision, the MLP…
iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer
Toshihiro Ota, Masato Taki
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer.…
Deep Residual Networks and Weight Initialization
Masato Taki
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of…