25 citations · 91 across the 19 of their papers we have counts for
20 papers
ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
Yijun Liang, Ming Li, Chenrui Fan +7
Color plays an important role in human perception and usually provides critical clues in visual reasoning. However, it is unclear whether and how vision-language models (VLMs) can…
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
Jiuhai Chen, Jianwei Yang, Haiping Wu +4
We present Florence-VL, a new family of multimodal large language models (MLLMs) with enriched visual representations produced by Florence-2, a generative vision foundation model.…
Multi-Objective Linguistic Control of Large Language Models
Dang Nguyen, Jiuhai Chen, Tianyi Zhou
Large language models (LLMs), despite their breakthroughs on many challenging benchmark tasks, lean to generate verbose responses and lack the controllability of output complexity,…
GenQA: Generating Millions of Instructions from a Handful of Prompts
Jiuhai Chen, Rifaa Qadri, Yuxin Wen +4
Most public instruction finetuning datasets are relatively small compared to the closed source datasets used to train industry models. To study questions about finetuning at scale,…
OPTune: Efficient Online Preference Tuning
Lichang Chen, Jiuhai Chen, Chenxi Liu +6
Reinforcement learning with human feedback~(RLHF) is critical for aligning Large Language Models (LLMs) with human preference. Compared to the widely studied offline version of RLH…
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang +8
Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks through vision instruction tuning on specific datasets. Howev…
ODIN: Disentangled Reward Mitigates Hacking in RLHF
Lichang Chen, Chen Zhu, Davit Soselia +6
In this work, we study the issue of reward hacking on the response length, a challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on LLMs. A well-formatted, verb…
Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements
Ming Li, Jiuhai Chen, Lichang Chen +1
Making LLMs speak for different, especially minority groups of people, and generate statements supporting their diverse or even controversial perspectives is critical to creating a…
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
Ming Li, Lichang Chen, Jiuhai Chen +3
Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the tra…
Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning
Ming Li, Lichang Chen, Jiuhai Chen +4
Recent advancements in Large Language Models (LLMs) have expanded the horizons of natural language understanding and generation. Notably, the output control and alignment with the…
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
Ming Li, Yong Zhang, Zhitao Li +6
In the realm of Large Language Models (LLMs), the balance between instruction data quality and quantity is a focal point. Recognizing this, we introduce a self-guided methodology f…
InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models
Lichang Chen, Jiuhai Chen, Tom Goldstein +2
Large language models~(LLMs) are instruction followers, but it can be challenging to find the best instruction for different situations, especially for black-box LLMs on which back…
How Many Demonstrations Do You Need for In-context Learning?
Jiuhai Chen, Lichang Chen, Chen Zhu +1
Large language models (LLMs) are capable to perform complex reasoning by in-context learning (ICL) when provided with a few input-output demonstrations (demos) and more powerful wh…
When do you need Chain-of-Thought Prompting for ChatGPT?
Jiuhai Chen, Lichang Chen, Heng Huang +1
Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think…
A Robust Stacking Framework for Training Deep Graph Models with Multifaceted Node Features
Jiuhai Chen, Jonas Mueller, Vassilis N. Ioannidis +2
Graph Neural Networks (GNNs) with numerical node features and graph structure as inputs have demonstrated superior performance on various supervised learning tasks with graph data.…
Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task
Jiuhai Chen, Chen Zhu, Bin Dai
Self-supervised learning (SSL) has achieved great success in a variety of computer vision tasks. However, the mechanism of how SSL works in these tasks remains a mystery. In this p…
Why Propagate Alone? Parallel Use of Labels and Features on Graphs
Yangkun Wang, Jiarui Jin, Weinan Zhang +7
Graph neural networks (GNNs) and label propagation represent two interrelated modeling strategies designed to exploit graph structure in tasks such as node property prediction. The…
Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features
Jiuhai Chen, Jonas Mueller, Vassilis N. Ioannidis +4
For supervised learning with tabular data, decision tree ensembles produced via boosting techniques generally dominate real-world applications involving iid training/test sets. How…
Particle-based Energetic Variational Inference
Yiwei Wang, Jiuhai Chen, Chun Liu +1
We introduce a new variational inference (VI) framework, called energetic variational inference (EVI). It minimizes the VI objective function based on a prescribed energy-dissipati…
Gaussian Process Assisted Active Learning of Physical Laws
Jiuhai Chen, Lulu Kang, Guang Lin
In many areas of science and engineering, discovering the governing differential equations from the noisy experimental data is an essential challenge. It is also a critical step in…