Predictability and Surprise in Large Generative Models

Deep Ganguli, Danny Hernandez, Liane Lovitt +27

Large-scale pre-training has recently emerged as a technique for creating capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. I…