765 citations · 2.5k across the 41 of their papers we have counts for
46 papers
MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions
Philipp Müller, Michal Balazia, Tobias Baur +9
Automatic analysis of human behaviour is a fundamental prerequisite for the creation of machines that can effectively interact with- and support humans in social interactions. In M…
Inferring Human Intentions from Predicted Action Probabilities
Lei Shi, Paul-Christian Bürkner, Andreas Bulling
Predicting the next action that a human is most likely to perform is key to human-AI collaboration and has consequently attracted increasing research interests in recent years. An…
Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning
Anna Penzkofer, Simon Schaefer, Florian Strohm +3
While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierar…
Exploring Natural Language Processing Methods for Interactive Behaviour Modelling
Guanhua Zhang, Matteo Bortoletto, Zhiming Hu +3
Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems…
Federated Learning for Appearance-based Gaze Estimation in the Wild
Mayar Elfares, Zhiming Hu, Pascal Reisert +2
Gaze estimation methods have significantly matured in recent years, but the large number of eye images required to train deep learning models poses significant privacy risks. In ad…
MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions
Philipp Müller, Michael Dietz, Dominik Schiller +5
Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, auto…
Neuro-Symbolic Visual Dialog
Adnen Abdessaied, Mihai Bâce, Andreas Bulling
We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significa…
Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
Ahmed Abdou, Ekta Sood, Philipp Müller +1
Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. spee…
VisRecall: Quantifying Information Visualisation Recallability via Question Answering
Yao Wang, Chuhan Jiao, Mihai Bâce +1
Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantita…
Scanpath Prediction on Information Visualisations
Yao Wang, Mihai Bâce, Andreas Bulling
We propose Unified Model of Saliency and Scanpaths (UMSS) -- a model that learns to predict visual saliency and scanpaths (i.e. sequences of eye fixations) on information visualisa…
Multimodal Integration of Human-Like Attention in Visual Question Answering
Ekta Sood, Fabian Kögel, Philipp Müller +3
Human-like attention as a supervisory signal to guide neural attention has shown significant promise but is currently limited to uni-modal integration - even for inherently multimo…
VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
Ekta Sood, Fabian Kögel, Florian Strohm +2
We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye t…
Neural Photofit: Gaze-based Mental Image Reconstruction
Florian Strohm, Ekta Sood, Sven Mayer +3
We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural netw…
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
Ekta Sood, Simon Tannert, Philipp Mueller +1
A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP). We propose…
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
Ekta Sood, Simon Tannert, Diego Frassinelli +2
While neural networks with attention mechanisms have achieved superior performance on many natural language processing tasks, it remains unclear to which extent learned attention r…
Adversarial Attacks on Classifiers for Eye-based User Modelling
Inken Hagestedt, Michael Backes, Andreas Bulling
An ever-growing body of work has demonstrated the rich information content available in eye movements for user modelling, e.g. for predicting users' activities, cognitive processes…
Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions
Mihai Bâce, Sander Staal, Andreas Bulling
Quantification of human attention is key to several tasks in mobile human-computer interaction (HCI), such as predicting user interruptibility, estimating noticeability of user int…
How far are we from quantifying visual attention in mobile HCI?
Mihai Bâce, Sander Staal, Andreas Bulling
With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a c…
Emergent Leadership Detection Across Datasets
Philipp Müller, Andreas Bulling
Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on sing…
SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements
Michael Xuelin Huang, Andreas Bulling
Recent methods to automatically calibrate stationary eye trackers were shown to effectively reduce inherent calibration distortion. However, these methods require additional inform…
Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications
Xucong Zhang, Yusuke Sugano, Andreas Bulling
Appearance-based gaze estimation methods that only require an off-the-shelf camera have significantly improved but they are still not yet widely used in the human-computer interact…
Moment-to-Moment Detection of Internal Thought from Eye Vergence Behaviour
Michae Xuelin Huang, Jiajia Li, Grace Ngai +2
Internal thought refers to the process of directing attention away from a primary visual task to internal cognitive processing. Internal thought is a pervasive mental activity and…
Privacy-Aware Eye Tracking Using Differential Privacy
Julian Steil, Inken Hagestedt, Michael Xuelin Huang +1
With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users' privacy is an ever more important, yet under-exp…
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Seonwook Park, Xucong Zhang, Andreas Bulling +1
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained re…