4 papers
SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations
Jason Wu, Shir-Kang Scott Jin, Yuyang Yuan +4
Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platfo…
Decoupling Vision and Language: Codebook Anchored Visual Adaptation
Jason Wu, Tianchen Zhao, Chang Liu +7
Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-spec…
ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources
Jason Wu, Yuyang Yuan, Kang Yang +2
Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute r…
MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT
Xiaomin Ouyang, Jason Wu, Tomoyoshi Kimura +4
Multimodal sensing systems are increasingly prevalent in various real-world applications. Most existing multimodal learning approaches heavily rely on training with a large amount…