3k citations · 5k across the 31 of their papers we have counts for
36 papers
Mask-Aware Policy Gradients for Diffusion Language Models
Haran Raajesh, Kulin Shah, Adam Klivans +1
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due t…
Cut Your Losses in Large-Vocabulary Language Models
Erik Wijmans, Brody Huval, Alexander Hertzberg +2
As language models grow ever larger, so do their vocabularies. This has shifted the memory footprint of LLMs during training disproportionately to one single layer: the cross-entro…
Predicting a Protein's Stability under a Million Mutations
Jeffrey Ouyang-Zhang, Daniel J. Diaz, Adam R. Klivans +1
Stabilizing proteins is a foundational step in protein engineering. However, the evolutionary pressure of all extant proteins makes identifying the scarce number of mutations that…
Long-tail Detection with Effective Class-Margins
Jang Hyun Cho, Philipp Krähenbühl
Large-scale object detection and instance segmentation face a severe data imbalance. The finer-grained object classes become, the less frequent they appear in our datasets. However…
Global Tracking Transformers
Xingyi Zhou, Tianwei Yin, Vladlen Koltun +1
We present a novel transformer-based architecture for global multi-object tracking. Our network takes a short sequence of frames as input and produces global trajectories for all o…
Learning from All Vehicles
Dian Chen, Philipp Krähenbühl
In this paper, we present a system to train driving policies from experiences collected not just from the ego-vehicle, but all vehicles that it observes. This system uses the behav…
Multimodal Virtual Point 3D Detection
Tianwei Yin, Xingyi Zhou, Philipp Krähenbühl
Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution…
Towards Long-Form Video Understanding
Chao-Yuan Wu, Philipp Krähenbühl
Our world offers a never-ending stream of visual stimuli, yet today's vision systems only accurately recognize patterns within a few seconds. These systems understand the present,…
Learning to drive from a world on rails
Dian Chen, Vladlen Koltun, Philipp Krähenbühl
We learn an interactive vision-based driving policy from pre-recorded driving logs via a model-based approach. A forward model of the world supervises a driving policy that predict…
Probabilistic two-stage detection
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl
We develop a probabilistic interpretation of two-stage object detection. We show that this probabilistic interpretation motivates a number of common empirical training practices. I…
Simple multi-dataset detection
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl
How do we build a general and broad object detection system? We use all labels of all concepts ever annotated. These labels span diverse datasets with potentially inconsistent taxo…
Center-based 3D Object Detection and Tracking
Tianwei Yin, Xingyi Zhou, Philipp Krähenbühl
Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with ad…
Lossless Image Compression through Super-Resolution
Sheng Cao, Chao-Yuan Wu, Philipp Krähenbühl
We introduce a simple and efficient lossless image compression algorithm. We store a low resolution version of an image as raw pixels, followed by several iterations of lossless su…
Tracking Objects as Points
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl
Tracking has traditionally been the art of following interest points through space and time. This changed with the rise of powerful deep networks. Nowadays, tracking is dominated b…
Learning by Cheating
Dian Chen, Brady Zhou, Vladlen Koltun +1
Vision-based urban driving is hard. The autonomous system needs to learn to perceive the world and act in it. We show that this challenging learning problem can be simplified by de…
Monocular Plan View Networks for Autonomous Driving
Dequan Wang, Coline Devin, Qi-Zhi Cai +2
Convolutions on monocular dash cam videos capture spatial invariances in the image plane but do not explicitly reason about distances and depth. We propose a simple transformation…
Does computer vision matter for action?
Brady Zhou, Philipp Krähenbühl, Vladlen Koltun
Computer vision produces representations of scene content. Much computer vision research is predicated on the assumption that these intermediate representations are useful for acti…
Objects as Points
Xingyi Zhou, Dequan Wang, Philipp Krähenbühl
Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each.…
Bottom-up Object Detection by Grouping Extreme and Center Points
Xingyi Zhou, Jiacheng Zhuo, Philipp Krähenbühl
With the advent of deep learning, object detection drifted from a bottom-up to a top-down recognition problem. State of the art algorithms enumerate a near-exhaustive list of objec…
Long-Term Feature Banks for Detailed Video Understanding
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan +3
To understand the world, we humans constantly need to relate the present to the past, and put events in context. In this paper, we enable existing video models to do the same. We p…
Joint Monocular 3D Vehicle Detection and Tracking
Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang +5
Vehicle 3D extents and trajectories are critical cues for predicting the future location of vehicles and planning future agent ego-motion based on those predictions. In this paper,…
Assessing Generalization in Deep Reinforcement Learning
Charles Packer, Katelyn Gao, Jernej Kos +3
Deep reinforcement learning (RL) has achieved breakthrough results on many tasks, but agents often fail to generalize beyond the environment they were trained in. As a result, deep…
Video Compression through Image Interpolation
Chao-Yuan Wu, Nayan Singhal, Philipp Krähenbühl
An ever increasing amount of our digital communication, media consumption, and content creation revolves around videos. We share, watch, and archive many aspects of our lives throu…
Compressed Video Action Recognition
Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu +3
Training robust deep video representations has proven to be much more challenging than learning deep image representations. This is in part due to the enormous size of raw video st…