Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms

Ene Meco, Emadeldeen Hamdan, A. Enis Cetin

To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (…