Transformers perform adaptive partial pooling

Vsevolod Kapatsinski

Any language model must decide what to say in novel contexts based on information from similar contexts. But what about contexts that are not novel but merely infrequent? In hierar…