42 citations · 57 across the 16 of their papers we have counts for
19 papers
MintAct: A Unified Visual Agent for Digital Environments
Mingfei Gao, Rui Tian, Haiming Gang +11
We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and…
OR-Action: Multi-Role Video Understanding with Fine-Grained Actions
Felix Tristram, Ege Özsoy, Christian Benz +3
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevai…
PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms
Tuna Gürbüz, Ege Özsoy, Tony Danjun Wang +1
Operating rooms (ORs) are cluttered, dynamic, highly occluded environments, where reliable spatial understanding is essential for situational awareness during complex surgical work…
Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting
Chantal Pellegrini, Adrian Delchev, Ege Özsoy +2
Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decis…
Recognizing Surgical Phases Anywhere: Few-Shot Test-time Adaptation and Task-graph Guided Refinement
Kun Yuan, Tingxuan Chen, Shi Li +9
The complexity and diversity of surgical workflows, driven by heterogeneous operating room settings, institutional protocols, and anatomical variability, present a significant chal…
Language Agents for Hypothesis-driven Clinical Decision Making with Reinforcement Learning
David Bani-Harouni, Chantal Pellegrini, Ege Özsoy +2
Clinical decision-making is a dynamic, interactive, and cyclic process where doctors have to repeatedly decide on which clinical action to perform and consider newly uncovered info…
EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records
Chantal Pellegrini, Ege Özsoy, David Bani-Harouni +2
Forecasting how a patient's condition is likely to evolve, including possible deterioration, recovery, treatment needs, and care transitions, could support more proactive and perso…
EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding
Ege Özsoy, Arda Mamur, Felix Tristram +4
Operating rooms (ORs) demand precise coordination among surgeons, nurses, and equipment in a fast-paced, occlusion-heavy environment, necessitating advanced perception models to en…
Specialized Foundation Models for Intelligent Operating Rooms
Ege Özsoy, Chantal Pellegrini, David Bani-Harouni +3
Surgical procedures unfold in complex environments demanding coordination between surgical teams, tools, imaging and increasingly, intelligent robotic systems. Ensuring safety and…
MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments
Ege Özsoy, Chantal Pellegrini, Tobias Czempiel +7
Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assista…
ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling
Ege Özsoy, Chantal Pellegrini, Matthias Keicher +1
Every day, countless surgeries are performed worldwide, each within the distinct settings of operating rooms (ORs) that vary not only in their setups but also in the personnel, too…
Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting
Chantal Pellegrini, Matthias Keicher, Ege Özsoy +1
Radiology reporting is a crucial part of the communication between radiologists and other medical professionals, but it can be time-consuming and error-prone. One approach to allev…
Whether and When does Endoscopy Domain Pretraining Make Sense?
Dominik Batić, Felix Holm, Ege Özsoy +2
Automated endoscopy video analysis is a challenging task in medical computer vision, with the primary objective of assisting surgeons during procedures. The difficulty arises from…
LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms
Ege Özsoy, Tobias Czempiel, Felix Holm +2
Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the op…
Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis
Chantal Pellegrini, Matthias Keicher, Ege Özsoy +3
Automated diagnosis prediction from medical images is a valuable resource to support clinical decision-making. However, such systems usually need to be trained on large amounts of…
Location-Free Scene Graph Generation
Ege Özsoy, Felix Holm, Mahdi Saleh +4
Scene Graph Generation (SGG) is a visual understanding task, aiming to describe a scene as a graph of entities and their relationships with each other. Existing works rely on locat…
CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
Chinedu Innocent Nwoye, Tong Yu, Saurav Sharma +46
Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. Th…
4D-OR: Semantic Scene Graphs for OR Domain Modeling
Ege Özsoy, Evin Pınar Örnek, Ulrich Eck +3
Surgical procedures are conducted in highly complex operating rooms (OR), comprising different actors, devices, and interactions. To date, only medically trained human experts are…
Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures
Ege Özsoy, Evin Pınar Örnek, Ulrich Eck +2
From a computer science viewpoint, a surgical domain model needs to be a conceptual one incorporating both behavior and data. It should therefore model actors, devices, tools, thei…