Converting MLPs into Polynomials in Closed Form

Nora Belrose, Alice Rigg

Recent work has shown that purely quadratic functions can replace MLPs in transformers with no significant loss in performance, while enabling new methods of interpretability based…