Supplementary Material: Implementation and Experiments for GAU-based Model

Zhenjie Liu

In February this year Google proposed a new Transformer variant called FLASH, which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a…