146 citations · 241 across the 21 of their papers we have counts for
22 papers
CUAWright: A Minimal Unified Interface for Digital Agents
Yadong Lu, Theodore Lee, Yifei Li +5
The prevailing approach to computer-use agents couples a model with a domain-specific harness: a browser or desktop environment equipped with human engineered tools that are fixed…
Fara-1.5: Scalable Learning Environments for Computer Use Agents
Ahmed Awadallah, Sahil Gupta, Yash Lara +12
Collecting computer use data from human demonstrations is expensive and slow, motivating the need for scalable generation strategies. This requires two key ingredients: environment…
Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging
Zeyi Liao, Yadong Lu, Boyu Gou +2
Graphical user interface (GUI) grounding, the process of mapping human instructions to GUI actions, serves as a fundamental basis to autonomous GUI agents. While existing grounding…
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
Vardaan Pahuja, Yadong Lu, Corby Rosset +5
Recent success in large multimodal models (LMMs) has sparked promising applications of agents capable of autonomously completing complex web tasks. While open-source LMM agents hav…
StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
Dilxat Muhtar, Yelong Shen, Yaming Yang +11
In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks directly from the given demonstrations without requiring gradient updates. While recent advances…
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Rogerio Bonatti, Dan Zhao, Francesco Bonacci +9
Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning…
Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype
Yadong Lu, Shitian Zhao, Boxiang Yun +4
Despite recent progress in enhancing the efficacy of Open-Domain Continual Learning (ODCL) in Vision-Language Models (VLM), failing to (1) correctly identify the Task-ID of a test…
OmniParser for Pure Vision Based GUI Agent
Yadong Lu, Jianwei Yang, Yelong Shen +1
The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal model…
AgentInstruct: Toward Generative Teaching with Agentic Flows
Arindam Mitra, Luciano Del Corro, Guoqing Zheng +11
Synthetic data is becoming increasingly important for accelerating the development of language models, both large and small. Despite several successful use cases, researchers also…
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
Liliang Ren, Yang Liu, Yadong Lu +3
Efficiently modeling sequences with infinite context length has long been a challenging problem. Previous approaches have either suffered from quadratic computational complexity or…
Multi-LoRA Composition for Image Generation
Ming Zhong, Yelong Shen, Shuohang Wang +6
Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated im…
Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models
Shitian Zhao, Zhuowan Li, Yadong Lu +2
While Multi-modal Language Models (MLMs) demonstrate impressive multimodal ability, they still struggle on providing factual and precise responses for tasks like visual question an…
Adapting LLM Agents with Universal Feedback in Communication
Kuan Wang, Yadong Lu, Michael Santacroce +3
Recent advances in large language models (LLMs) have demonstrated potential for LLM agents. To facilitate the training for these agents with both linguistic feedback and non-lingui…
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Yadong Lu, Chunyuan Li, Haotian Liu +3
Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open…
Efficient RLHF: Reducing the Memory Usage of PPO
Michael Santacroce, Yadong Lu, Han Yu +2
Reinforcement Learning with Human Feedback (RLHF) has revolutionized language modeling by aligning models with human preferences. However, the RL stage, Proximal Policy Optimizatio…
In-Context Learning Unlocked for Diffusion Models
Zhendong Wang, Yifan Jiang, Yadong Lu +5
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to…
Resolving Extreme Jet Substructure
Yadong Lu, Alexis Romero, Michael James Fenton +2
We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to ). Previous studies…
Progressive Neural Image Compression with Nested Quantization and Latent Ordering
Yadong Lu, Yinhao Zhu, Yang Yang +2
We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstrea…
SARM: Sparse Autoregressive Model for Scalable Generation of Sparse Images in Particle Physics
Yadong Lu, Julian Collado, Daniel Whiteson +1
Generation of simulated data is essential for data analysis in particle physics, but current Monte Carlo methods are very computationally expensive. Deep-learning-based generative…
Differentiable Joint Pruning and Quantization for Hardware Efficiency
Ying Wang, Yadong Lu, Tijmen Blankevoort
We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between mo…
Towards Automating Precision Studies of Clone Detectors
Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu +5
Current research in clone detection suffers from poor ecosystems for evaluating precision of clone detection tools. Corpora of labeled clones are scarce and incomplete, making eval…
Oreo: Detection of Clones in the Twilight Zone
Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu +2
Source code clones are categorized into four types of increasing difficulty of detection, ranging from purely textual (Type-1) to purely semantic (Type-4). Most clone detectors rep…