Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Abstract

The interaction between the vestibular and ocular system has primarily been studied in controlled environments. Consequently, off-the shelf tools for categorization of gaze events (e.g. fixations, pursuits, saccade) fail when head movements are allowed. Our approach was to collect a novel, naturalistic, and multimodal dataset of eye+head movements when subjects performed everyday tasks while wearing a mobile eye tracker equipped with an inertial measurement unit and a 3D stereo camera. This Gaze-in-the-Wild dataset (GW) includes eye+head rotational velocities (deg/s), infrared eye images and scene imagery (RGB+D). A portion was labelled by coders into gaze motion events with a mutual agreement of 0.72 sample based Cohen's $\kappa$. This labelled data was used to train and evaluate two machine learning algorithms, Random Forest and a Recurrent Neural Network model, for gaze event classification. Assessment involved the application of established and novel event based performance metrics. Classifiers achieve $\sim$90$\%$ human performance in detecting fixations and saccades but fall short (60$\%$) on detecting pursuit movements. Moreover, pursuit classification is far worse in the absence of head movement information. A subsequent analysis of feature significance in our best-performing model revealed a reliance upon absolute eye and head velocity, indicating that classification does not require spatial alignment of the head and eye tracking coordinate systems. The GW dataset, trained classifiers and evaluation metrics will be made publicly available with the intention of facilitating growth in the emerging area of head-free gaze event classification.

arXiv comment23 pages, 11 figures, 10 tables, Dataset can be found at http://www.cis.rit.edu/~rsk3900/

References in corpus (5)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
• End-to-End Eye Movement Detection Using Convolutional Neural Networks
• Bayesian Identification of Fixations, Saccades, and Smooth Pursuits

Cited by in corpus (7)

• TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types
• An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Automatic Gaze Analysis: A Survey of Deep Learning based Approaches
• RIT-Eyes: Rendering of near-eye images for eye-tracking applications
• LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images
• Data-driven Head Motion Generation through Natural Gaze-Head Coordination