Sparse Attention Decomposition Applied to Circuit Tracing

Gabriel Franco, Mark Crovella

Many papers have shown that attention heads work in conjunction with each other to perform complex tasks. It's frequently assumed that communication between attention heads is via…