Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Zihan Qiu, Zeyu Huang, Youcheng Huang +1

The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empiri…