6 papers
Steerability of Instrumental-Convergence Tendencies in LLMs
Jakub Hoscilowicz
We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is wh…
Adversarial Confusion Attack: Disrupting Multimodal Large Language Models
Jakub Hoscilowicz, Artur Janicki
We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is…
TinyClick: Single-Turn Agent for Empowering GUI Automation
Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz +4
We present an UI agent for user interface (UI) interaction tasks, using Vision-Language Model Florence-2-Base. The agent's primary task is identifying the screen coordinates of the…
Large Language Models as Carriers of Hidden Messages
Jakub Hoscilowicz, Pawel Popiolek, Jan Rudkowski +2
Simple fine-tuning can embed hidden text into large language models (LLMs), which is revealed only when triggered by a specific query. Applications include LLM fingerprinting, wher…
ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz +2
With the growing reliance on digital devices equipped with graphical user interfaces (GUIs), such as computers and smartphones, the need for effective automation tools has become i…
Non-Linear Inference Time Intervention: Improving LLM Truthfulness
Jakub Hoscilowicz, Adam Wiacek, Jan Chojnacki +4
In this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information. We further developed the Inference…