most citedWhat Context Features Can Transformer Language Models Use?

5 citations · 5 across the 1 of their papers we have counts for

collaborators

1 paper

cs.CL2021★ 5 cited

What Context Features Can Transformer Language Models Use?

Joe O'Connor, Jacob Andreas

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model pre…