Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-produc…