52 citations · 139 across the 17 of their papers we have counts for
17 papers
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
Zineng Tang, Ziyi Yang, Mahmoud Khademi +3
We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL)…
Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization
Tanmay Gautam, Reid Pryzant, Ziyi Yang +2
Vector Quantization (VQ) is a well-known technique in deep learning for extracting informative discrete latent representations. VQ-embedded models have shown impressive results in…
i-Code Studio: A Configurable and Composable Framework for Integrative AI
Yuwei Fang, Mahmoud Khademi, Chenguang Zhu +8
Artificial General Intelligence (AGI) requires comprehensive understanding and generation capabilities for a variety of tasks spanning different modalities and functionalities. Int…
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data
Ziyi Yang, Mahmoud Khademi, Yichong Xu +16
The convergence of text, visual, and audio data is a key step towards human-like artificial intelligence, however the current Vision-Language-Speech landscape is dominated by encod…
Any-to-Any Generation via Composable Diffusion
Zineng Tang, Ziyi Yang, Chenguang Zhu +2
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any comb…
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
Soumya Sanyal, Yichong Xu, Shuohang Wang +5
Logical reasoning of text is an important ability that requires understanding the information present in the text, their interconnections, and then reasoning through them to infer…
Unifying Vision, Text, and Layout for Universal Document Processing
Zineng Tang, Ziyi Yang, Guoxin Wang +6
We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including docume…
Empowering Language Models with Knowledge Graph Reasoning for Question Answering
Ziniu Hu, Yichong Xu, Wenhao Yu +5
Answering open-domain questions requires world knowledge about in-context entities. As pre-trained Language Models (LMs) lack the power to store all required knowledge, external kn…
UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization
Yulong Chen, Yang Liu, Ruochen Xu +4
The high annotation costs and diverse demands of various summarization tasks motivate the development of few-shot summarization. However, despite the emergence of many summarizatio…
MACSum: Controllable Summarization with Mixed Attributes
Yusen Zhang, Yang Liu, Ziyi Yang +6
Controllable summarization allows users to generate customized summaries with specified attributes. However, due to the lack of designated annotations of controlled summaries, exis…
Global Contrastive Batch Sampling via Optimization on Sample Permutations
Vin Sachidananda, Ziyi Yang, Chenguang Zhu
Contrastive Learning has recently achieved state-of-the-art performance in a wide range of tasks. Many contrastive learning approaches use mined hard negatives to make batches more…
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Zhenhailong Wang, Manling Li, Ruochen Xu +10
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question…
Automatic Rule Induction for Interpretable Semi-Supervised Learning
Reid Pryzant, Ziyi Yang, Yichong Xu +2
Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Meanwhile, pretrained transformer models act as black-box correl…
Anomaly Detection with Domain Adaptation
Ziyi Yang, Iman Soltani Bozchalooi, Eric Darve
We study the problem of semi-supervised anomaly detection with domain adaptation. Given a set of normal data from a source domain and a limited amount of normal examples from a tar…
Filtered Inner Product Projection for Crosslingual Embedding Alignment
Vin Sachidananda, Ziyi Yang, Chenguang Zhu
Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space. Recently, t…
Leveraging Lead Bias for Zero-shot Abstractive News Summarization
Chenguang Zhu, Ziyi Yang, Robert Gmyr +2
A typical journalistic convention in news articles is to deliver the most salient information in the beginning, also known as the lead bias. While this phenomenon can be exploited…
Language Distribution Prediction based on Batch Markov Monte Carlo Simulation with Migration
XingYu Fu, ZiYi Yang, XiuWen Duan
Language spreading is a complex mechanism that involves issues like culture, economics, migration, population etc. In this paper, we propose a set of methods to model the dynamics…