194 citations · 1.1k across the 147 of their papers we have counts for
29 papers · 1 filter
LiFT: Loop Flow Transformers
Mohammad Mahdi Derakhshani, Pedro M. P. Curvo, Gertjan J. Burghouts +2
We introduce Loop Flow Transformers (LiFT), a family of looped generative models that scales computation by repeatedly applying a shared Diffusion Transformer (DiT) core, with only…
GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning
Jie Ou, Shuaihong Jiang, Yingjun Du +1
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, DoRA, and HiRA, enable lightweight adaptation of large pre-trained models via low-rank updates. However, existing PEFT…
Structured-Noise Masked Modeling for Video, Audio and Beyond
Aritra Bhowmik, Fida Mohammad Thoker, Carlos Hinojosa +2
Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the structural properties of different da…
GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts
Sameer Ambekar, Zehao Xiao, Xiantong Zhen +1
We consider the problem of test-time domain generalization, where a model is trained on several source domains and adjusted on target domains never seen during training. Different…
DynaPrompt: Dynamic Test-Time Prompt Tuning
Zehao Xiao, Shilin Yan, Jack Hong +6
Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt…
Beyond Model Adaptation at Test Time: A Survey
Zehao Xiao, Cees G. M. Snoek
Machine learning algorithms have achieved remarkable success across various disciplines, use cases and applications, under the prevailing assumption that training and test samples…
Prompt Diffusion Robustifies Any-Modality Prompt Learning
Yingjun Du, Gaowen Liu, Yuzhang Shang +3
Foundation models enable prompt-based classifiers for zero-shot and few-shot learning. Nonetheless, the conventional method of employing fixed prompts suffers from distributional s…
IPO: Interpretable Prompt Optimization for Vision-Language Models
Yingjun Du, Wenfang Sun, Cees G. M. Snoek
Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input tex…
Graph Neural Networks for Learning Equivariant Representations of Neural Networks
Miltiadis Kofinas, Boris Knyazev, Yan Zhang +5
Neural networks that process the parameters of other neural networks find applications in domains as diverse as classifying implicit neural representations, generating neural netwo…
Data Augmentations in Deep Weight Spaces
Aviv Shamsian, David W. Zhang, Aviv Navon +10
Learning in weight spaces, where neural networks process the weights of other deep neural networks, has emerged as a promising research direction with applications in various field…
Probabilistic Test-Time Generalization by Variational Neighbor-Labeling
Sameer Ambekar, Zehao Xiao, Jiayi Shen +2
This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. We follow the strict separation…
EMO: Episodic Memory Optimization for Few-Shot Meta-Learning
Yingjun Du, Jiayi Shen, Xiantong Zhen +1
Few-shot meta-learning presents a challenge for gradient descent optimization due to the limited number of training samples per task. To address this issue, we propose an episodic…
ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion
Yingjun Du, Zehao Xiao, Shengcai Liao +1
Prototype-based meta-learning has emerged as a powerful technique for addressing few-shot learning challenges. However, estimating a deterministic prototype using a simple average…
MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Tasks
Wenfang Sun, Yingjun Du, Xiantong Zhen +3
Meta-learning algorithms are able to learn a new task using previously learned knowledge, but they often require a large number of meta-training tasks which may not be readily avai…
Energy-Based Test Sample Adaptation for Domain Generalization
Zehao Xiao, Xiantong Zhen, Shengcai Liao +1
In this paper, we propose energy-based sample adaptation at test time for domain generalization. Where previous works adapt their models to target domains, we adapt the unseen targ…
Unlocking Slot Attention by Changing Optimal Transport Costs
Yan Zhang, David W. Zhang, Simon Lacoste-Julien +2
Slot attention is a powerful method for object-centric modeling in images and videos. However, its set-equivariance limits its ability to handle videos with a dynamic number of obj…
Variational Model Perturbation for Source-Free Domain Adaptation
Mengmeng Jing, Xiantong Zhen, Jingjing Li +1
We aim for source-free domain adaptation, where the task is to deploy a model pre-trained on source domains to target domains. The challenges stem from the distribution shift from…
Learning to Generalize across Domains on Single Test Samples
Zehao Xiao, Xiantong Zhen, Ling Shao +1
We strive to learn a model from a set of source domains that generalizes well to unseen target domains. The main challenge in such a domain generalization scenario is the unavailab…
Generative Kernel Continual learning
Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao +1
Kernel continual learning by \citet{derakhshani2021kernel} has recently emerged as a strong continual learner due to its non-parametric ability to tackle task interference and cata…
Hierarchical Variational Memory for Few-shot Learning Across Domains
Yingjun Du, Xiantong Zhen, Ling Shao +1
Neural memory enables fast adaptation to new tasks with just a few training samples. Existing memory models store features only from the single last layer, which does not generaliz…
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
Yan Zhang, David W. Zhang, Simon Lacoste-Julien +2
Most set prediction models in deep learning use set-equivariant operations, but they actually operate on multisets. We show that set-equivariant functions cannot represent certain…
Kernel Continual Learning
Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao +1
This paper introduces kernel continual learning, a simple but effective variant of continual learning that leverages the non-parametric nature of kernel methods to tackle catastrop…
Pruning Edges and Gradients to Learn Hypergraphs from Larger Sets
David W. Zhang, Gertjan J. Burghouts, Cees G. M. Snoek
This paper aims for set-to-hypergraph prediction, where the goal is to infer the set of relations for a given set of entities. This is a common abstraction for applications in part…
MetaKernel: Learning Variational Random Features with Limited Labels
Yingjun Du, Haoliang Sun, Xiantong Zhen +4
Few-shot learning deals with the fundamental and challenging problem of learning from a few annotated samples, while being able to generalize well on new tasks. The crux of few-sho…