228 citations · 1.5k across the 123 of their papers we have counts for
28 papers · 1 filter
Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction
Shivang Chopra, Fotis Iliopoulos, Zsolt Kira +1
Click-Through Rate prediction, a core task in recommendation and advertising systems, relies on modeling interactions among sparse categorical features. Explicit cross networks are…
Directional Gradient Projection for Robust Fine-Tuning of Foundation Models
Chengyue Huang, Junjiao Tian, Brisa Maneechotesuwan +2
Robust fine-tuning aims to adapt large foundation models to downstream tasks while preserving their robustness to distribution shifts. Existing methods primarily focus on constrain…
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia +6
We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typica…
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
Junjiao Tian, Chengyue Huang, Zsolt Kira
Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is be…
RenderBender: A Survey on Adversarial Attacks Using Differentiable Rendering
Matthew Hull, Haoran Wang, Matthew Lau +8
Differentiable rendering techniques like Gaussian Splatting and Neural Radiance Fields have become powerful tools for generating high-fidelity models of 3D objects and scenes. Thei…
ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI
Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya +4
Intelligent embodied agents need to quickly adapt to new scenarios by integrating long histories of experience into decision-making. For instance, a robot in an unfamiliar house in…
Reinforcement Learning via Auxiliary Task Distillation
Abhinav Narayan Harish, Larry Heck, Josiah P. Hanna +2
We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems…
Grounding Multimodal Large Language Models in Actions
Andrew Szot, Bogdan Mazoure, Harsh Agrawal +3
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM…
Adaptive Memory Replay for Continual Learning
James Seale Smith, Lazar Valkov, Shaunak Halbe +4
Foundation Models (FMs) have become the hallmark of modern AI, however, these models are trained on massive data, leading to financially expensive training. Updating FMs as new dat…
ConstraintMatch for Semi-constrained Clustering
Jann Goschenhofer, Bernd Bischl, Zsolt Kira
Constrained clustering allows the training of classification models using pairwise constraints only, which are weak and relatively easy to mine, while still yielding full-supervisi…
Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models
Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov +3
The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. Though the generative process is traditionally understood as an…
Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra +3
We present the task of "Social Rearrangement", consisting of cooperative everyday tasks like setting up the dinner table, tidying a house or unpacking groceries in a simulated mult…
Training Energy-Based Normalizing Flow with Score-Matching Objectives
Chen-Hao Chao, Wei-Fang Sun, Yen-Chang Hsu +2
In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy…
BC-IRL: Learning Generalizable Reward Functions from Demonstrations
Andrew Szot, Amy Zhang, Dhruv Batra +2
How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy obje…
System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games
Indranil Sur, Zachary Daniels, Abrar Rahman +16
As Artificial and Robotic Systems are increasingly deployed and relied upon for real-world applications, it is important that they exhibit the ability to continually learn and adap…
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning
James Seale Smith, Paola Cascante-Bonilla, Assaf Arbelle +7
Recently, large-scale pre-trained Vision-and-Language (VL) foundation models have demonstrated remarkable capabilities in many zero-shot downstream tasks, achieving competitive res…
FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization
Junjiao Tian, James Seale Smith, Zsolt Kira
Federated Learning (FL) seeks to distribute model training across local clients without collecting data in a centralized data-center, hence removing data-privacy concerns. A major…
A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
Yen-Chang Hsu, James Smith, Yilin Shen +2
Knowledge distillation (KD) is a substantial strategy for transferring learned knowledge from one neural network model to another. A vast number of methods have been developed for…
A Closer Look at Rehearsal-Free Continual Learning
James Seale Smith, Junjiao Tian, Shaunak Halbe +2
Continual learning is a setting where machine learning models learn novel concepts from continuously shifting training data, while simultaneously avoiding degradation of knowledge…
Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
Junjiao Tian, Yen-Change Hsu, Yilin Shen +2
Moving beyond testing on in-distribution data works on Out-of-Distribution (OOD) detection have recently increased in popularity. A recent attempt to categorize OOD data introduces…
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
Andrew Szot, Alex Clegg, Eric Undersander +18
We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contr…
Posterior Re-calibration for Imbalanced Datasets
Junjiao Tian, Yen-Cheng Liu, Nathan Glaser +2
Neural Networks can perform poorly when the training label distribution is heavily imbalanced, as well as when the testing data differs from the training distribution. In order to…
Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
Chia-Wen Kuo, Chih-Yao Ma, Jia-Bin Huang +1
Recent advances in semi-supervised learning methods rely on estimating the categories of unlabeled data using a model trained on the labeled data (pseudo-labeling) and using the un…
Leveraging Semantics for Incremental Learning in Multi-Relational Embeddings
Angel Daruna, Weiyu Liu, Zsolt Kira +1
Service robots benefit from encoding information in semantically meaningful ways to enable more robust task execution. Prior work has shown multi-relational embeddings can encode s…