67 citations · 143 across the 16 of their papers we have counts for
17 papers
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
Andrea Burns, Krishna Srinivasan, Joshua Ainslie +5
Webpages have been a rich resource for language and vision-language tasks. Yet only pieces of webpages are kept: image-caption pairs, long text articles, or raw HTML, never all in…
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences
David Uthus, Santiago Ontañón, Joshua Ainslie +1
We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architec…
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Andrea Burns, Krishna Srinivasan, Joshua Ainslie +5
Webpages have been a rich, scalable resource for vision-language and language only tasks. Yet only pieces of webpages are kept in existing datasets: image-caption pairs, long text…
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
Haoxuan You, Mandy Guo, Zhecan Wang +3
The field of vision and language has witnessed a proliferation of pre-trained foundation models. Most existing methods are independently pre-trained with contrastive objective like…
CoLT5: Faster Long-Range Transformers with Conditional Computation
Joshua Ainslie, Tao Lei, Michiel de Jong +9
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but…
LongT5: Efficient Text-To-Text Transformer for Long Sequences
Mandy Guo, Joshua Ainslie, David Uthus +4
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we pre…