285 citations · 493 across the 26 of their papers we have counts for
26 papers
ViCo: Engaging Video Comment Generation with Human Preference Rewards
Yuchong Sun, Bei Liu, Xu Chen +2
Engaging video comments play an important role in video social media, as they are the carrier of feelings, thoughts, or humor of the audience. Preliminary works have made initial e…
Transferring Foundation Models for Generalizable Robotic Manipulation
Jiange Yang, Wenhui Tan, Chuhao Jin +6
Improving the generalization capabilities of general-purpose robotic manipulation agents in the real world has long been a significant challenge. Existing approaches often rely on…
User Behavior Simulation with Large Language Model based Agents
Lei Wang, Jingsen Zhang, Hao Yang +11
Simulating high quality user behavior data has always been a fundamental problem in human-centered applications, where the major difficulty originates from the intricate mechanism…
AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation
Chuhao Jin, Wenhui Tan, Jiange Yang +4
We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks. These tasks often invol…
ComedicSpeech: Text To Speech For Stand-up Comedies in Low-Resource Scenarios
Yuyue Wang, Huan Xiao, Yihan Wu +1
Text to Speech (TTS) models can generate natural and high-quality speech, but it is not expressive enough when synthesizing speech with dramatic expressiveness, such as stand-up co…
Text2Poster: Laying out Stylized Texts on Retrieved Images
Chuhao Jin, Hongteng Xu, Ruihua Song +1
Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience. In this paper,…
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Hongpeng Lin, Ludan Ruan, Wenke Xia +8
To facilitate the research on intelligent and human-like chatbots with multi-modal context, we introduce a new video-based multi-modal dialogue dataset, called TikTalk. We collect…
TeViS:Translating Text Synopses to Video Storyboards
Xu Gu, Yuchong Sun, Feiyue Ni +5
A video storyboard is a roadmap for video creation which consists of shot-by-shot images to visualize key plots in a text synopsis. Creating video storyboards, however, remains cha…
VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
Yihan Wu, Junliang Guo, Xu Tan +7
Video dubbing aims to translate the original speech in a film or television program into the speech in a target language, which can be achieved with a cascaded system consisting of…
Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Yuchong Sun, Hongwei Xue, Ruihua Song +3
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on shor…
Multi-Modal Experience Inspired AI Creation
Qian Cao, Xu Chen, Ruihua Song +3
AI creation, such as poem or lyrics generation, has attracted increasing attention from both industry and academic communities, with many promising models proposed in the past few…
Self-supervised Context-aware Style Representation for Expressive Speech Synthesis
Yihan Wu, Xi Wang, Shaofei Zhang +3
Expressive speech synthesis, like audiobook synthesis, is still challenging for style representation learning and prediction. Deriving from reference audio or predicting style tags…
A Roadmap for Big Model
Sha Yuan, Hanyu Zhao, Shuai Zhao +97
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm. Researchers have achieved various outcomes in the c…
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios
Yihan Wu, Xu Tan, Bohan Li +5
Adaptive text to speech (TTS) can synthesize new voices in zero-shot scenarios efficiently, by using a well-trained source TTS model without adapting it on the speech data of new s…
Class-aware Sounding Objects Localization via Audiovisual Correspondence
Di Hu, Yake Wei, Rui Qian +3
Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achie…
Towards artificial general intelligence via a multimodal foundation model
Nanyi Fei, Zhiwu Lu, Yizhao Gao +9
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods hav…
Stylistic Retrieval-based Dialogue System with Unparallel Training Data
Hao Fu, Yan Wang, Ruihua Song +2
The ability of a dialog system to express consistent language style during conversations has a direct, positive impact on its usability and on user satisfaction. Although previous…
Pre-Trained Models: Past, Present and Future
Xu Han, Zhengyan Zhang, Ning Ding +21
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophis…
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
Yuqi Huo, Manli Zhang, Guangzhen Liu +32
Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction bet…
ScriptWriter: Narrative-Guided Script Generation
Yutao Zhu, Ruihua Song, Zhicheng Dou +2
It is appealing to have a system that generates a story or scripts automatically from a story-line, even though this is still out of our reach. In dialogue systems, it would also b…
"Love is as Complex as Math": Metaphor Generation System for Social Chatbot
Danning Zheng, Ruihua Song, Tianran Hu +2
As the wide adoption of intelligent chatbot in human daily life, user demands for such systems evolve from basic task-solving conversations to more casual and friend-like communica…
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
Shizhe Chen, Bei Liu, Jianlong Fu +6
A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products. In this paper, we tackle…
From Text to Sound: A Preliminary Study on Retrieving Sound Effects to Radio Stories
Songwei Ge, Curtis Xuan, Ruihua Song +3
Sound effects play an essential role in producing high-quality radio stories but require enormous labor cost to add. In this paper, we address the problem of automatically adding s…
Image Inspired Poetry Generation in XiaoIce
Wen-Feng Cheng, Chao-Chung Wu, Ruihua Song +3
Vision is a common source of inspiration for poetry. The objects and the sentimental imprints that one perceives from an image may lead to various feelings depending on the reader.…