31 citations · 66 across the 30 of their papers we have counts for
30 papers
A Versatile Multimodal Agent for Multimedia Content Generation
Daoan Zhang, Wenlin Yao, Xiaoyang Wang +3
With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and…
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
Hongliang He, Wenlin Yao, Kaixin Ma +5
The rapid development of large language and multimodal models has sparked significant interest in using proprietary models, such as GPT-4o, to develop autonomous agents capable of…
Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization
Daoan Zhang, Guangchen Lan, Dong-Jun Han +8
Existing post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement learning (RL) methods; the former is stable during training but suffer…
DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
Murong Yue, Wenlin Yao, Haitao Mi +3
Enhancing the capability of large language models (LLMs) in reasoning has gained significant attention in recent years. Previous studies have demonstrated the effectiveness of vari…
DeFine: Decision-Making with Analogical Reasoning over Factor Profiles
Yebowen Hu, Xiaoyang Wang, Wenlin Yao +5
LLMs are ideal for decision-making thanks to their ability to reason over long contexts. However, challenges arise when processing speech transcripts that describe complex scenario…
HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows
Wenlin Yao, Haitao Mi, Dong Yu
Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still lim…
DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
Liqiang Jing, Zhehui Huang, Xiaoyang Wang +6
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents…
When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives
Yebowen Hu, Kaiqiang Song, Sangwoo Cho +5
Reasoning is most powerful when an LLM accurately aggregates relevant information. We examine the critical role of information aggregation in reasoning by requiring the LLM to anal…
MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions
Zhenwen Liang, Dian Yu, Wenhao Yu +4
Large language models (LLMs) have demonstrated impressive capabilities in mathematical problem solving, particularly in single turn question answering formats. However, real world…
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
Xinran Zhao, Hongming Zhang, Xiaoman Pan +4
For a LLM to be trustworthy, its confidence level should be well-calibrated with its actual performance. While it is now common sense that LLM performances are greatly impacted by…
InFoBench: Evaluating Instruction Following Ability in Large Language Models
Yiwei Qin, Kaiqiang Song, Yebowen Hu +7
This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions. Addressing a gap…
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Hongliang He, Wenlin Yao, Kaixin Ma +5
The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in…
TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
Shuyi Xie, Wenlin Yao, Yong Dai +11
Large language models (LLMs) have shown impressive capabilities across various natural language tasks. However, evaluating their alignment with human preferences remains a challeng…
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao +5
With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of…
From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning
Xuansheng Wu, Wenlin Yao, Jianshu Chen +4
Large Language Models (LLMs) have achieved remarkable success, where instruction tuning is the critical step in aligning LLMs with user intentions. In this work, we investigate how…
A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
Neeraj Varshney, Wenlin Yao, Hongming Zhang +2
Recently developed large language models have achieved remarkable success in generating fluent and coherent text. However, these models often tend to 'hallucinate' which critically…
MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning
Zhenwen Liang, Dian Yu, Xiaoman Pan +4
Reasoning in mathematical domains remains a significant challenge for relatively small language models (LMs). Many current methods focus on specializing LMs in mathematical reasoni…
Thrust: Adaptively Propels Large Language Models with External Knowledge
Xinran Zhao, Hongming Zhang, Xiaoman Pan +3
Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, maki…
Could Small Language Models Serve as Recommenders? Towards Data-centric Cold-start Recommendations
Xuansheng Wu, Huachi Zhou, Yucheng Shi +3
Recommendation systems help users find matched items based on their previous behaviors. Personalized recommendation becomes challenging in the absence of historical user-item inter…
Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations
James Y. Huang, Wenlin Yao, Kaiqiang Song +3
Traditional sentence embedding models encode sentences into vector representations to capture useful properties such as the semantic similarity between sentences. However, in addit…
ZeroKBC: A Comprehensive Benchmark for Zero-Shot Knowledge Base Completion
Pei Chen, Wenlin Yao, Hongming Zhang +4
Knowledge base completion (KBC) aims to predict the missing links in knowledge graphs. Previous KBC tasks and approaches mainly focus on the setting where all test entities and rel…
NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization
Chao Zhao, Faeze Brahman, Kaiqiang Song +3
Narrative summarization aims to produce a distilled version of a narrative to describe its most salient events and characters. Summarizing a narrative is challenging as it requires…
Efficient Zero-shot Event Extraction with Context-Definition Alignment
Hongming Zhang, Wenlin Yao, Dong Yu
Event extraction (EE) is the task of identifying interested event mentions from text. Conventional efforts mainly focus on the supervised setting. However, these supervised models…
Salience Allocation as Guidance for Abstractive Summarization
Fei Wang, Kaiqiang Song, Hongming Zhang +6
Abstractive summarization models typically learn to capture the salient information from scratch implicitly. Recent literature adds extractive summaries as guidance for abstractive…