1.1k citations · 1.8k across the 50 of their papers we have counts for
54 papers
Less Language, More Latents: Annotation-Efficient VLAs for Driving
Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania
Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructio…
Conformalized Large Language Models under Configuration Shift
Yuqicheng Zhu, Jialin Yu, Lin Li +7
Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets…
Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
Kemal Oksuz, Alexandru Buburuzan, Anthony Knittel +2
The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choice…
Modeling Gene Expression Distributional Shifts for Unseen Genetic Perturbations
Kalyan Ramakrishnan, Jonathan G. Hedley, Sisi Qu +5
We train a neural network to predict distributional responses in gene expression following genetic perturbations. This is an essential task in early-stage drug discovery, where suc…
Mixture of Experts Made Intrinsically Interpretable
Xingyi Yang, Constantin Venhoff, Ashkan Khakzar +4
Neurons in large language models often exhibit \emph{polysemanticity}, simultaneously encoding multiple unrelated concepts and obscuring interpretability. Instead of relying on pos…
AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
Magnus Sesodia, Rilton Franzone, Alina Petrova +7
Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challengin…
MObI: Multimodal Object Inpainting Using Diffusion Models
Alexandru Buburuzan, Anuj Sharma, John Redford +2
Safety-critical applications, such as autonomous driving, require extensive multimodal data for rigorous testing. Methods based on synthetic data are gaining prominence due to the…
What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz +4
Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via…
On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines
Selim Kuzucu, Kemal Oksuz, Jonathan Sadeghi +1
Reliable usage of object detectors require them to be calibrated -- a crucial problem that requires careful attention. Recent approaches towards this involve (1) designing new loss…
Placing Objects in Context via Inpainting for Out-of-distribution Segmentation
Pau de Jorge, Riccardo Volpi, Puneet K. Dokania +2
When deploying a semantic segmentation model into the real world, it will inevitably encounter semantic classes that were not seen during training. To ensure a safe deployment of s…
Random Representations Outperform Online Continually Learned Representations
Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru +3
Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the e…
Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentation
Francisco Eiras, Kemal Oksuz, Adel Bibi +2
Referring Image Segmentation (RIS) - the problem of identifying objects in images through natural language sentences - is a challenging task currently mostly solved through supervi…
MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection
Kemal Oksuz, Selim Kuzucu, Tom Joy +1
Combining the strengths of many existing predictors to obtain a Mixture of Experts which is superior to its individual components is an effective way to improve the performance wit…
Fine-tuning can cripple your foundation model; preserving features may be the solution
Jishnu Mukhoti, Yarin Gal, Philip H. S. Torr +1
Pre-trained foundation models, due to their enormous capacity and exposure to vast amounts of data during pre-training, are known to have learned plenty of real-world concepts. An…
Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration
Kemal Oksuz, Tom Joy, Puneet K. Dokania
The current approach for testing the robustness of object detectors suffers from serious deficiencies such as improper methods of performing out-of-distribution detection and using…
Graph Inductive Biases in Transformers without Message Passing
Liheng Ma, Chen Lin, Derek Lim +5
Transformers for graph data are increasingly widely studied and successful in numerous learning tasks. Graph inductive biases are crucial for Graph Transformers, and previous works…
Online Continual Learning Without the Storage Constraint
Ameya Prabhu, Zhipeng Cai, Puneet Dokania +3
Traditional online continual learning (OCL) research has primarily focused on mitigating catastrophic forgetting with fixed and limited storage allocation throughout an agent's lif…
Computationally Budgeted Continual Learning: What Does Matter?
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania +4
Continual Learning (CL) aims to sequentially train models on streams of incoming data that vary in distribution by preserving previous knowledge while adapting to new data. Current…
Raising the Bar on the Evaluation of Out-of-Distribution Detection
Jishnu Mukhoti, Tsung-Yu Lin, Bor-Chun Chen +4
In image classification, a lot of development has happened in detecting out-of-distribution (OoD) data. However, most OoD detection methods are evaluated on a standard set of datas…
Query-based Hard-Image Retrieval for Object Detection at Test Time
Edward Ayers, Jonathan Sadeghi, John Redford +2
There is a longstanding interest in capturing the error behaviour of object detectors by finding images where their performance is likely to be unsatisfactory. In real-world applic…
An Impartial Take to the CNN vs Transformer Robustness Contest
Francesco Pinto, Philip H. S. Torr, Puneet K. Dokania
Following the surge of popularity of Transformers in Computer Vision, several studies have attempted to determine whether they could be more robust to distribution shifts and provi…
Sample-dependent Adaptive Temperature Scaling for Improved Calibration
Tom Joy, Francesco Pinto, Ser-Nam Lim +2
It is now well known that neural networks can be wrong with high confidence in their predictions, leading to poor calibration. The most common post-hoc approach to compensate for t…
RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness
Francesco Pinto, Harry Yang, Ser-Nam Lim +2
We show that the effectiveness of the well celebrated Mixup [Zhang et al., 2018] can be further improved if instead of using it as the sole learning objective, it is utilized as an…
Catastrophic overfitting can be induced with discriminative non-robust features
Guillermo Ortiz-Jiménez, Pau de Jorge, Amartya Sanyal +5
Adversarial training (AT) is the de facto method for building robust neural networks, but it can be computationally expensive. To mitigate this, fast single-step attacks can be use…