23 citations · 97 across the 42 of their papers we have counts for
43 papers
SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar +6
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR aro…
Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer Workspace
Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar +8
An audio language model is a black box in a specific way: we see what it says, never what it works out on the way there, and chain-of-thought monitoring helps only if the model wri…
MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding
Shuai Wang, Wangyuan Ding, Yixian Shen +5
Recent vision-language models demonstrate impressive general visual understanding, yet their art interpretation remains shallow: they describe surface content but struggle with for…
Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs
Pengyu Zhang, Klim Zaporojets, Congfeng Cao +2
Multi-Modal Knowledge Graphs (MMKGs) enrich entities with multiple modalities such as text and images, yet entities with highly similar multi-modal features remain difficult to dis…
Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoning
Yixian Shen, Zhiheng Yang, Qi Bi +6
Multimodal spatial reasoning often relies on long chains of intermediate textual and visual thoughts, where accumulating visual tokens and dense cross-modal attention incur substan…
A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
Shuai Wang, Hongyi Zhu, Jia-Hong Huang +6
Understanding artworks requires multi-step reasoning over visual content and cultural, historical, and stylistic context. While recent multimodal large language models show promise…
A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech
Jia-Hong Huang, Seulgi Kim, Yi Chieh Liu +5
Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker id…
Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment
Pengyu Zhang, Klim Zaporojets, Jie Liu +2
Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., lo…
Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding
Jiaqi Li, Shuntian Zheng, Yixian Shen +4
Video Temporal Grounding (VTG) localizes the temporal boundaries of query-relevant moments in long, untrimmed videos, making video-language-model prohibitively expensive. While rec…
DeepEyeNet: Generating Medical Report for Retinal Images
Jia-Hong Huang
The increasing prevalence of retinal diseases poses a significant challenge to the healthcare system, as the demand for ophthalmologists surpasses the available workforce. This imb…
MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
Yixian Shen, Qi Bi, Jia-Hong Huang +3
We present a new adaptation method MaCP, Minimal yet Mighty adaptive Cosine Projection, that achieves exceptional performance while requiring minimal parameters and memory for fine…
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation
Yixian Shen, Qi Bi, Jia-Hong Huang +3
Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM). However, it still encounters…
Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
Jia-Hong Huang, Yixian Shen, Hongyi Zhu +2
Large Language Models (LLMs) have shown remarkable performance across various tasks, but the escalating demands on computational resources pose significant challenges, particularly…
Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models
Jia-Hong Huang, Hongyi Zhu, Yixian Shen +2
Evaluating the quality of automatically generated image descriptions is a complex task that requires metrics capturing various dimensions, such as grammaticality, coverage, accurac…
MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
Yixian Shen, Qi Bi, Jia-Hong Huang +3
We present a new adaptation method MaCP, Minimal yet Mighty adaptive Cosine Projection, that achieves exceptional performance while requiring minimal parameters and memory for fine…
Personalized Video Summarization using Text-Based Queries and Conditional Modeling
Jia-Hong Huang
The proliferation of video content on platforms like YouTube and Vimeo presents significant challenges in efficiently locating relevant information. Automatic video summarization a…
A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation
Weijia Zhang, Mohammad Aliannejadi, Jiahuan Pei +3
Large language models (LLMs) often generate content with unsupported or unverifiable content, known as "hallucinations." To address this, retrieval-augmented LLMs are employed to i…
Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization using Large Language Models
Weijia Zhang, Jia-Hong Huang, Svitlana Vakulenko +3
Query-focused summarization (QFS) is a fundamental task in natural language processing with broad applications, including search engines and report generation. However, traditional…
Automated Retinal Image Analysis and Medical Report Generation through Deep Learning
Jia-Hong Huang
The increasing prevalence of retinal diseases poses a significant challenge to the healthcare system, as the demand for ophthalmologists surpasses the available workforce. This imb…
A Novel Evaluation Framework for Image2Text Generation
Jia-Hong Huang, Hongyi Zhu, Yixian Shen +3
Evaluating the quality of automatically generated image descriptions is challenging, requiring metrics that capture various aspects such as grammaticality, coverage, correctness, a…
Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models
Jia-Hong Huang, Chao-Chun Yang, Yixian Shen +2
The legal landscape encompasses a wide array of lawsuit types, presenting lawyers with challenges in delivering timely and accurate information to clients, particularly concerning…
Ada-HGNN: Adaptive Sampling for Scalable Hypergraph Neural Networks
Shuai Wang, David W. Zhang, Jia-Hong Huang +4
Hypergraphs serve as an effective model for depicting complex connections in various real-world scenarios, from social to biological networks. The development of Hypergraph Neural…
Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics
Weijia Zhang, Mohammad Aliannejadi, Yifei Yuan +3
Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations." To mitigate this, retrieval-augmented LLMs incorporate citations, groundi…
QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs
Weijia Zhang, Vaishali Pal, Jia-Hong Huang +2
Table summarization is a crucial task aimed at condensing information from tabular data into concise and comprehensible textual summaries. However, existing approaches often fall s…