STLM Engineering Report: Dropout

Dylan Hillier, Leon Guertler, Bobby Cheng +1

In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance first…