241 citations · 465 across the 16 of their papers we have counts for
4 papers · 1 filter
Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
Moya Chen, Paul A. Crook, Stephen Roller
We demonstrate that large language models are able to simulate Task Oriented Dialogues in novel domains, provided only with an API implementation and a list of goals. We show these…
Hash Layers For Large Sparse Models
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam +1
We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforwar…
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
Staircase Attention for Recurrent Processing of Sequences
Da Ju, Stephen Roller, Sainbayar Sukhbaatar +1
Attention mechanisms have become a standard tool for sequence modeling tasks, in particular by stacking self-attention layers over the entire input sequence as in the Transformer a…