7 citations · 9 across the 13 of their papers we have counts for
15 papers
Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng +10
Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precis…
HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
Hei Yi Mak, Shadan Golestan, Hoang Le +10
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-…
HiFloat4 Format for Language Model Pre-training on Ascend NPUs
Mehran Taghian, Yunke Peng, Xing Huang +22
Training large foundation models at low numerical precision is one of the most promising directions for reducing the compute and memory cost of modern AI. Recent 4-bit floating-poi…
SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding
Tanzila Rahman, Renjie Liao, Leonid Sigal
Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, an…
To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
Jiayun Luo, Wan-Cyuan Fan, Lyuyang Wang +4
Large Vision Language Models (LVLMs) have recently emerged as powerful architectures capable of understanding and reasoning over both visual and textual information. These models t…
MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
Wan-Cyuan Fan, Tanzila Rahman, Leonid Sigal
With advances in foundational and vision-language models, and effective fine-tuning techniques, a large number of both general and special-purpose models have been developed for a…
Visual Concept-driven Image Generation with Text-to-Image Diffusion Model
Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee +3
Text-to-image (TTI) diffusion models have demonstrated impressive results in generating high-resolution images of complex and imaginative scenes. Recent approaches have further ext…
Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi +1
The quality of the prompts provided to text-to-image diffusion models determines how faithful the generated content is to the user's intent, often requiring `prompt engineering'. T…
Make-A-Story: Visual Memory Conditioned Consistent Story Generation
Tanzila Rahman, Hsin-Ying Lee, Jian Ren +3
There has been a recent explosion of impressive generative models that can produce high quality images (or videos) conditioned on text descriptions. However, all such approaches re…
TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation
Tanzila Rahman, Mengyu Yang, Leonid Sigal
The recent success of transformer models in language, such as BERT, has motivated the use of such architectures for multi-modal feature learning and tasks. However, most multi-moda…
Weakly-supervised Audio-visual Sound Source Detection and Separation
Tanzila Rahman, Leonid Sigal
Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from arti…
An Improved Attention for Visual Question Answering
Tanzila Rahman, Shih-Han Chou, Leonid Sigal +1
We consider the problem of Visual Question Answering (VQA). Given an image and a free-form, open-ended, question, expressed in natural language, the goal of VQA system is to provid…
Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning
Tanzila Rahman, Bicheng Xu, Leonid Sigal
Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from lang…
Convolutional Temporal Attention Model for Video-based Person Re-identification
Tanzila Rahman, Mrigank Rochan, Yang Wang
The goal of video-based person re-identification is to match two input videos, so that the distance of the two videos is small if two videos contain the same person. A common appro…
Video-based Person Re-identification Using Spatial-Temporal Attention Networks
Shivansh Rao, Tanzila Rahman, Mrigank Rochan +1
We consider the problem of video-based person re-identification. The goal is to identify a person from videos captured under different cameras. In this paper, we propose an efficie…