584 citations · 688 across the 17 of their papers we have counts for
4 papers · 1 filter
A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks
William Merrill, Nikolaos Tsilivis, Aman Shukla
Grokking is a phenomenon where a model trained on an algorithmic task first overfits but, then, after a large amount of additional training, undergoes a phase transition to general…
A Logic for Expressing Log-Precision Transformers
William Merrill, Ashish Sabharwal
One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et a…
Extracting Finite Automata from RNNs Using State Merging
William Merrill, Nikolaos Tsilivis
One way to interpret the behavior of a blackbox recurrent neural network (RNN) is to extract from it a more interpretable discrete computational model, like a finite state machine,…
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
William Merrill, Vivek Ramanujan, Yoav Goldberg +2
The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the trainin…