106 citations · 248 across the 15 of their papers we have counts for
16 papers
Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing
Peiyuan Zhang, Guoqiang Wei, Yilong Zhao +7
We present VSA2, a frontier trainable sparse attention for video DiTs. VSA2 includes a variety of new architectural features and training procedures that we apply across all stages…
FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanations
Yan Zeng, Changlu Guo, Oskar Kristoffersen +3
Visual counterfactual explanations aim to change classifier decisions through realistic and localized edits while preserving decision-irrelevant content. Existing DDPM-based method…
Selective formation of metastable polymorphs in solid-state synthesis
Yan Zeng, Nathan J. Szymanski, Tanjin He +6
Metastable polymorphs often result from the interplay between thermodynamics and kinetics. Despite advances in predictive synthesis for solution-based techniques, there remains a l…
What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?
Yan Zeng, Hanbo Zhang, Jiani Zheng +5
Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the…
eTag: Class-Incremental Learning with Embedding Distillation and Task-Oriented Generation
Libo Huang, Yan Zeng, Chuanguang Yang +3
Class-Incremental Learning (CIL) aims to solve the neural networks' catastrophic forgetting problem, which refers to the fact that once the network updates on a new task, its perfo…
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks
Xinsong Zhang, Yan Zeng, Jipeng Zhang +1
Foundation models or pre-trained models have substantially improved the performance of various language, vision, and vision-language understanding tasks. However, existing foundati…
X-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
Yan Zeng, Xinsong Zhang, Hang Li +3
Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others u…
EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
Tiannan Wang, Wangchunshu Zhou, Yan Zeng +1
Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of par…
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
Yan Zeng, Wangchunshu Zhou, Ao Luo +2
In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architec…
VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
Wangchunshu Zhou, Yan Zeng, Shizhe Diao +1
Recent advances in vision-language pre-training (VLP) have demonstrated impressive performance in a range of vision-language (VL) tasks. However, there exist several challenges for…
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
Yan Zeng, Xinsong Zhang, Hang Li
Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted feat…
A probabilistic deep learning approach to automate the interpretation of multi-phase diffraction spectra
Nathan J. Szymanski, Christopher J. Bartel, Yan Zeng +2
Autonomous synthesis and characterization of inorganic materials requires the automatic and accurate analysis of X-ray diffraction spectra. For this task, we designed a probabilist…
Open-Domain Dialogue Generation Based on Pre-trained Language Models
Yan Zeng, Jian-Yun Nie
Pre-trained language models have been successfully used in response generation for open-domain dialogue. Four main frameworks have been proposed: (1) Transformer-ED using Transform…
Multi-Domain Dialogue State Tracking based on State Graph
Yan Zeng, Jian-Yun Nie
We investigate the problem of multi-domain Dialogue State Tracking (DST) with open vocabulary, which aims to extract the state from the dialogue. Existing approaches usually concat…
Jointly Optimizing State Operation Prediction and Value Generation for Dialogue State Tracking
Yan Zeng, Jian-Yun Nie
We investigate the problem of multi-domain Dialogue State Tracking (DST) with open vocabulary. Existing approaches exploit BERT encoder and copy-based RNN decoder, where the encode…
A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation
Yan Zeng, Jian-Yun Nie
Conditioned dialogue generation suffers from the scarcity of labeled responses. In this work, we exploit labeled non-dialogue text data related to the condition, which are much eas…