86 citations · 543 across the 30 of their papers we have counts for
34 papers
Assessing the State of Self-Supervised Human Activity Recognition using Wearables
Harish Haresamudram, Irfan Essa, Thomas Plötz
The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the…
Neural Design Network: Graphic Layout Generation with Constraints
Hsin-Ying Lee, Lu Jiang, Irfan Essa +4
Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs. Layout generation differs from pixel-level image synthesis and…
Novel evaluation of surgical activity recognition models using task-based efficiency metrics
Aneeq Zia, Liheng Guo, Linlin Zhou +2
Purpose: Surgical task-based metrics (rather than entire procedure metrics) can be used to improve surgeon training and, ultimately, patient care through focused training intervent…
Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
K. Niranjan Kumar, Irfan Essa, Sehoon Ha +1
We explore the problem of estimating the mass distribution of an articulated object by an interactive robotic agent. Our method predicts the mass distribution of an object by using…
Floors are Flat: Leveraging Semantics for Real-Time Surface Normal Prediction
Steven Hickson, Karthik Raveendran, Alireza Fathi +2
We propose 4 insights that help to significantly improve the performance of deep learning models that predict surface normals and semantic labels from a single RGB image. These ins…
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
Erik Wijmans, Samyak Datta, Oleksandr Maksymets +6
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answe…
Audio-Visual Scene-Aware Dialog
Huda Alamri, Vincent Cartillier, Abhishek Das +9
We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history…
Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
Jonathan C Balloch, Varun Agrawal, Irfan Essa +1
Robot perception systems need to perform reliable image segmentation in real-time on noisy, raw perception data. State-of-the-art segmentation approaches use large CNN models and c…
Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition
Unaiza Ahsan, Rishi Madhok, Irfan Essa
We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self-supervised approaches have used spatial context…
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
Chiori Hori, Huda Alamri, Jue Wang +10
Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-worl…
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
Huda Alamri, Vincent Cartillier, Raphael Gontijo Lopes +8
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-…
Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learning
Aneeq Zia, Andrew Hung, Irfan Essa +1
Adverse surgical outcomes are costly to patients and hospitals. Approaches to benchmark surgical care are often limited to gross measures across the entire procedure despite the pe…
Efficient Hierarchical Graph-Based Segmentation of RGBD Videos
Steven Hickson, Stan Birchfield, Irfan Essa +1
We present an efficient and scalable algorithm for segmenting 3D RGBD point clouds by combining depth, color, and temporal information using a multistage, hierarchical graph-based…
Let's Dance: Learning From Online Dance Videos
Daniel Castro, Steven Hickson, Patsorn Sangkloy +4
In recent years, deep neural network approaches have naturally extended to the video domain, in their simplest case by aggregating per-frame classifications as a baseline for actio…
DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks
Unaiza Ahsan, Chen Sun, Irfan Essa
We propose an action recognition framework using Gen- erative Adversarial Networks. Our model involves train- ing a deep convolutional generative adversarial network (DCGAN) using…
Object category learning and retrieval with weak supervision
Steven Hickson, Anelia Angelova, Irfan Essa +1
We consider the problem of retrieving objects from image data and learning to classify them into meaningful semantic categories with minimal supervision. To that end, we propose a…
Automated Surgical Skill Assessment in RMIS Training
Aneeq Zia, Irfan Essa
Purpose: Manual feedback in basic RMIS training can consume a significant amount of time from expert surgeons' schedule and is prone to subjectivity. While VR-based training tasks…
One-Shot Learning for Semantic Segmentation
Amirreza Shaban, Shray Bansal, Zhen Liu +2
Low-shot learning methods for image classification support learning from sparse data. We extend these techniques to support dense semantic image segmentation. Specifically, we trai…
Semantic Instance Labeling Leveraging Hierarchical Segmentation
Steven Hickson, Irfan Essa, Henrik Christensen
Most of the approaches for indoor RGBD semantic la- beling focus on using pixels or superpixels to train a classi- fier. In this paper, we implement a higher level segmentation usi…
Eyemotion: Classifying facial expressions in VR using eye-tracking cameras
Steven Hickson, Nick Dufour, Avneesh Sud +2
One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and ther…
Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment
Aneeq Zia, Yachna Sharma, Vinay Bettadapura +2
Purpose: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save exper…
Complex Event Recognition from Images with Few Training Examples
Unaiza Ahsan, Chen Sun, James Hays +1
We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples. We introduce a novel framework to discover event…
Leveraging Contextual Cues for Generating Basketball Highlights
Vinay Bettadapura, Caroline Pantofaru, Irfan Essa
The massive growth of sports videos has resulted in a need for automatic generation of sports highlights that are comparable in quality to the hand-edited highlights produced by br…
Discovering Picturesque Highlights from Egocentric Vacation Videos
Vinay Bettadapura, Daniel Castro, Irfan Essa
We present an approach for identifying picturesque highlights from large amounts of egocentric video data. Given a set of egocentric videos captured over the course of a vacation,…