Publications (13)
Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang +10
Multi-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the…
Learning Structured Reasoning via Tractable Trajectory Control
Po-Nien Kung, Zhen Yang, Jeffrey Luo +7
Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning t…
LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planning
Silin Meng, Yiwei Wang, Cheng-Fu Yang +2
Path planning is a fundamental scientific problem in robotics and autonomous navigation, requiring the derivation of efficient routes from starting to destination points while avoi…
Scene Graph Expansion for Semantics-Guided Image Outpainting
Chiao-An Yang, Cheng-Yo Tan, Wan-Cyuan Fan +3
In this paper, we address the task of semantics-guided image outpainting, which is to complete an image by generating semantically practical content. Different from most existing i…
Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang +9
We introduce Perception Encoder Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Built on PE, PE-AV makes se…
Customize Multi-modal RAI Guardrails with Precedent-based predictions
Cheng-Fu Yang, Thanh Tran, Christos Christodoulopoulos +3
A multi-modal guardrail must effectively filter image content based on user-defined policies, identifying material that may be hateful, reinforce harmful stereotypes, contain expli…
Re-ReST: Reflection-Reinforced Self-Training for Language Agents
Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu +2
Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impra…
Target-Free Text-guided Image Manipulation
Wan-Cyuan Fan, Cheng-Fu Yang, Chiao-An Yang +1
We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground tru…
Learning to Rank Chain-of-Thought: Using a Small Model
Eric Hanchen Jiang, Haozheng Luo, Shengyuan Pang +9
Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy O…
Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty
Cheng-Fu Yang, Haoyang Xu, Te-Lin Wu +3
Task planning for embodied AI has been one of the most challenging problems where the community does not meet a consensus in terms of formulation. In this paper, we aim to tackle t…
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following
Cheng-Fu Yang, Yen-Chun Chen, Jianwei Yang +4
End-to-end Transformers have demonstrated an impressive success rate for Embodied Instruction Following when the environment has been seen in training. However, they tend to strugg…
Paraphrasing Is All You Need for Novel Object Captioning
Cheng-Fu Yang, Yao-Hung Hubert Tsai, Wan-Cyuan Fan +3
Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. Due to the absence of caption annotation, ca…
Verbalized Representation Learning for Interpretable Few-Shot Generalization
Cheng-Fu Yang, Da Yin, Wenbo Hu +4
Humans recognize objects after observing only a few examples, a remarkable capability enabled by their inherent language understanding of the real-world environment. Developing ver…