30 citations · 64 across the 13 of their papers we have counts for
14 papers
Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders
Samuel Stevens, Jacob Beattie, Tanya Berger-Wolf +2
Foundation models in several scientific domains, including visual domains, learn representations that capture complex semantics from their respective fields. Despite this, most exi…
BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
Ziheng Zhang, Xinyue Ma, Arpita Chowdhury +9
This work investigates descriptive captions as an additional source of supervision for biological multimodal foundation models. Images and captions can be viewed as complementary s…
BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning
Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo +13
Foundation models trained at scale exhibit remarkable emergent behaviors, learning new capabilities beyond their initial training objectives. We find such emergent behaviors in bio…
Interpretable and Testable Vision Features via Sparse Autoencoders
Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf +1
To truly understand vision models, we must not only interpret their learned features but also validate these interpretations through controlled experiments. While earlier work offe…
Towards Artwork Explanation in Large-scale Vision Language Models
Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito +2
Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating capabilities in text generation and comprehension. However, it has not been clari…
BioCLIP: A Vision Foundation Model for the Tree of Life
Samuel Stevens, Jiaman Wu, Matthew J Thompson +9
Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion…
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Xiang Yue, Yuansheng Ni, Kai Zhang +19
We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU…
A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
Dipanjyoti Paul, Arpita Chowdhury, Xinqi Xiong +11
We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate cla…
Roll Up Your Sleeves: Working with a Collaborative and Engaging Task-Oriented Dialogue System
Lingbo Mo, Shijie Chen, Ziru Chen +10
We introduce TacoBot, a user-centered task-oriented digital assistant designed to guide users through complex real-world tasks with multiple steps. Covering a wide range of cooking…
Mind2Web: Towards a Generalist Agent for the Web
Xiang Deng, Yu Gu, Boyuan Zheng +5
We introduce Mind2Web, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website.…
Memorization for Good: Encryption with Autoregressive Language Models
Samuel Stevens, Yu Su
Over-parameterized neural language models (LMs) can memorize and recite long sequences of training data. While such memorization is normally associated with undesired properties su…
arXivEdits: Understanding the Human Revision Process in Scientific Writing
Chao Jiang, Wei Xu, Samuel Stevens
Scientific publications are the primary means to communicate research discoveries, where the writing quality is of crucial importance. However, prior work studying the human editin…
Bootstrapping a User-Centered Task-Oriented Dialogue System
Shijie Chen, Ziru Chen, Xiang Deng +8
We present TacoBot, a task-oriented dialogue system built for the inaugural Alexa Prize TaskBot Challenge, which assists users in completing multi-step cooking and home improvement…
An Investigation of Language Model Interpretability via Sentence Editing
Samuel Stevens, Yu Su
Pre-trained language models (PLMs) like BERT are being used for almost all language-related tasks, but interpreting their behavior still remains a significant challenge and many im…