112 citations · 559 across the 26 of their papers we have counts for
27 papers
Differentially Private Video Activity Recognition
Zelun Luo, Yuliang Zou, Yijin Yang +6
In recent years, differential privacy has seen significant advancements in image classification; however, its application to video activity recognition remains under-explored. This…
PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees
Chulin Xie, De-An Huang, Wenda Chu +4
Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high…
ISB: Image-to-Image Schrödinger Bridge
Guan-Horng Liu, Arash Vahdat, De-An Huang +3
We propose Image-to-Image Schrödinger Bridge (ISB), a new class of conditional diffusion models that directly learn the nonlinear diffusion processes between two given distribu…
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Zhuolin Yang, Wei Ping, Zihan Liu +13
Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-art results in image-to-text generation. However, these models stor…
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Manli Shu, Weili Nie, De-An Huang +4
Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on ha…
Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding
Stephen Su, Samuel Kwong, Qingyu Zhao +3
There has been an increasing interest in multi-task learning for video understanding in recent years. In this work, we propose a generalized notion of multi-task learning by incorp…
MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
De-An Huang, Zhiding Yu, Anima Anandkumar
We propose MinVIS, a minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedur…
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Linxi Fan, Guanzhi Wang, Yunfan Jiang +7
Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manuall…
Auditing AI models for Verified Deployment under Semantic Specifications
Homanga Bharadhwaj, De-An Huang, Chaowei Xiao +2
Auditing trained deep learning (DL) models prior to deployment is vital for preventing unintended consequences. One of the biggest challenges in auditing is the lack of human-inter…
PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks
Jiankai Sun, De-An Huang, Bo Lu +3
In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the…
SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
Linxi Fan, Guanzhi Wang, De-An Huang +4
Generalization has been a long-standing challenge for reinforcement learning (RL). Visual RL, in particular, can be easily distracted by irrelevant factors in high-dimensional obse…
Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
Boxiao Pan, Haoye Cai, De-An Huang +4
Video captioning is a challenging task that requires a deep understanding of visual scenes. State-of-the-art methods generate captions using either scene-level or object-level info…
Motion Reasoning for Goal-Based Imitation Learning
De-An Huang, Yu-Wei Chao, Chris Paxton +5
We address goal-based imitation learning, where the aim is to output the symbolic goal from a third-person video demonstration. This enables the robot to plan for execution and rep…
Regression Planning Networks
Danfei Xu, Roberto Martín-Martín, De-An Huang +3
Recent learning-to-plan methods have shown promising results on planning directly from observation space. Yet, their ability to plan for long-horizon tasks is limited by the accura…
Imitation Learning for Human Pose Prediction
Borui Wang, Ehsan Adeli, Hsu-kuang Chiu +2
Modeling and prediction of human motion dynamics has long been a challenging problem in computer vision, and most existing methods rely on the end-to-end supervised training of var…
Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning
De-An Huang, Danfei Xu, Yuke Zhu +4
We address one-shot imitation learning, where the goal is to execute a previously unseen task based on a single demonstration. While there has been exciting progress in this direct…
Procedure Planning in Instructional Videos
Chien-Yi Chang, De-An Huang, Danfei Xu +3
In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in every…
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
Chien-Yi Chang, De-An Huang, Yanan Sui +2
We address weakly supervised action alignment and segmentation in videos, where only the order of occurring actions is available during training. We propose Discriminative Differen…
Action-Agnostic Human Pose Forecasting
Hsu-kuang Chiu, Ehsan Adeli, Borui Wang +2
Predicting and forecasting human dynamics is a very interesting but challenging task with several prospective applications in robotics, health-care, etc. Recently, several methods…
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
De-An Huang, Suraj Nair, Danfei Xu +5
Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to…
Learning to Decompose and Disentangle Representations for Video Prediction
Jun-Ting Hsieh, Bingbin Liu, De-An Huang +2
Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the high-dimensionality…
Visual Forecasting by Imitating Dynamics in Natural Sequences
Kuo-Hao Zeng, William B. Shen, De-An Huang +2
We introduce a general framework for visual forecasting, which directly imitates visual sequences without additional supervision. As a result, our model can be applied at several s…
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
De-An Huang, Joseph J. Lim, Li Fei-Fei +1
We propose an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity (e.g., "dressing") to the action (e.g., "mix yogu…
Unsupervised Learning of Long-Term Motion Dynamics for Videos
Zelun Luo, Boya Peng, De-An Huang +2
We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns…