584 citations · 688 across the 17 of their papers we have counts for
4 papers · 1 filter
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
William Merrill, Vivek Ramanujan, Yoav Goldberg +2
The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the trainin…
CORD-19: The COVID-19 Open Research Dataset
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar +25
The COVID-19 Open Research Dataset (CORD-19) is a growing resource of scientific papers on COVID-19 and related historical coronavirus research. CORD-19 is designed to facilitate t…
A Formal Hierarchy of RNN Architectures
William Merrill, Gail Weiss, Yoav Goldberg +3
We develop a formal hierarchy of the expressive capacity of RNN architectures. The hierarchy is based on two formal properties: space complexity, which measures the RNN's memory, a…
On the Linguistic Capacity of Real-Time Counter Automata
William Merrill
Counter machines have achieved a newfound relevance to the field of natural language processing (NLP): recent work suggests some strong-performing recurrent neural networks utilize…