402 citations · 611 across the 25 of their papers we have counts for
28 papers
On Locality and Length Generalization in Visual Reasoning
Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3
A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes…
Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh +5
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large lang…
Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks
Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya +1
Many dexterous manipulation tasks are non-markovian in nature, yet little attention has been paid to this fact in the recent upsurge of the vision-language-action (VLA) paradigm. A…
Generative Scenario Rollouts for End-to-End Autonomous Driving
Rajeev Yasarla, Deepti Hegde, Shizhong Han +10
Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation lear…
Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Apratim Bhattacharyya, Bicheng Xu, Sanjay Haresh +6
Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI a…
RoCA: Robust Cross-Domain End-to-End Autonomous Driving
Rajeev Yasarla, Shizhong Han, Hsin-Pai Cheng +7
End-to-end (E2E) autonomous driving has recently emerged as a new paradigm, offering significant potential. However, few studies have looked into the practical challenge of deploym…
Can Vision-Language Models Answer Face to Face Questions in the Real-World?
Reza Pourreza, Rishit Dagli, Apratim Bhattacharyya +3
AI models have made significant strides in recent years in their ability to describe and answer questions about real-world images. They have also made progress in the ability to co…
Enhancing Hallucination Detection through Noise Injection
Litian Liu, Reza Pourreza, Sunny Panchal +4
Large Language Models (LLMs) are prone to generating plausible yet incorrect responses, known as hallucinations. Effectively detecting hallucinations is therefore crucial for the s…
Distilling Multi-modal Large Language Models for Autonomous Driving
Deepti Hegde, Rajeev Yasarla, Hong Cai +7
Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as…
ClevrSkills: Compositional Language and Visual Reasoning in Robotics
Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya +1
Robotics tasks are highly compositional by nature. For example, to perform a high-level task like cleaning the table a robot must employ low-level capabilities of moving the effect…
What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction
Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger +10
Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e…
Painter: Teaching Auto-regressive Language Models to Draw Sketches
Reza Pourreza, Apratim Bhattacharyya, Sunny Panchal +3
Large language models (LLMs) have made tremendous progress in natural language understanding and they have also been successfully adopted in other domains such as computer vision,…
Look, Remember and Reason: Grounded reasoning in videos with language models
Apratim Bhattacharyya, Sunny Panchal, Mingu Lee +3
Multi-modal language models (LM) have recently shown promising performance in high-level reasoning tasks on videos. However, existing methods still fall short in tasks like causal…
KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients
Niklas Hanselmann, Katrin Renz, Kashyap Chitta +2
Simulators offer the possibility of safe, low-cost development of self-driving systems. However, current driving simulators exhibit naïve behavior models for background traffic. Ha…
Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centers
Apratim Bhattacharyya, Daniel Olmeda Reino, Mario Fritz +1
Accurate prediction of pedestrian and bicyclist paths is integral to the development of reliable autonomous vehicles in dense urban environments. The interactions between vehicle a…
Haar Wavelet based Block Autoregressive Flows for Trajectories
Apratim Bhattacharyya, Christoph-Nikolas Straehle, Mario Fritz +1
Prediction of trajectories such as that of pedestrians is crucial to the performance of autonomous agents. While previous works have leveraged conditional generative models like GA…
Normalizing Flows with Multi-Scale Autoregressive Priors
Shweta Mahajan, Apratim Bhattacharyya, Mario Fritz +2
Flow-based generative models are an important class of exact inference models that admit efficient inference and sampling for image synthesis. Owing to the efficiency constraints o…
"Best-of-Many-Samples" Distribution Matching
Apratim Bhattacharyya, Mario Fritz, Bernt Schiele
Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders…
Conditional Flow Variational Autoencoders for Structured Sequence Prediction
Apratim Bhattacharyya, Michael Hanselmann, Mario Fritz +2
Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for s…
SampleFix: Learning to Generate Functionally Diverse Fixes
Hossein Hajipour, Apratim Bhattacharyya, Cristian-Alexandru Staicu +1
Automatic program repair holds the potential of dramatically improving the productivity of programmers during the software development process and correctness of software in genera…
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Ahmed Salem, Apratim Bhattacharya, Michael Backes +2
Machine learning (ML) has progressed rapidly during the past decade and the major factor that drives such development is the unprecedented large-scale data. As data generation is a…
Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods
Apratim Bhattacharyya, Mario Fritz, Bernt Schiele
For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence. In real-world scenarios, future states become in…
Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective
Apratim Bhattacharyya, Bernt Schiele, Mario Fritz
For autonomous agents to successfully operate in the real world, anticipation of future events and states of their environment is a key competence. This problem has been formalized…
Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
Apratim Bhattacharyya, Mario Fritz, Bernt Schiele
For autonomous agents to successfully operate in the real world, anticipation of future events and states of their environment is a key competence. This problem can be formalized a…