Normalized Architectures are Natively 4-Bit

Maxim Fishman, Brian Chmiel, Ron Banner +2

Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hype…