45 citations · 93 across the 25 of their papers we have counts for
4 papers · 1 filter
Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
Satoki Ishikawa, Rio Yokota, Ryo Karakida
Local learning, which trains a network through layer-wise local targets and losses, has been studied as an alternative to backpropagation (BP) in neural computation. However, its a…
Optimal Layer Selection for Latent Data Augmentation
Tomoumi Takase, Ryo Karakida
While data augmentation (DA) is generally applied to input data, several studies have reported that applying DA to hidden layers in neural networks, i.e., feature augmentation, can…
Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking
Ryo Karakida, Toshihiro Ota, Masato Taki
Transformers have established themselves as the leading neural network model in natural language processing and are increasingly foundational in various domains. In vision, the MLP…
Self-attention Networks Localize When QK-eigenspectrum Concentrates
Han Bao, Ryuichiro Hataya, Ryo Karakida
The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the degree…