FixationNet: Forecasting Eye Fixations in Task-Oriented Virtual Environments

TVCG Best Journal Nominees Award

Human Eye Fixation Analysis and Forecasting in Task-Oriented Virtual Environments

Zhiming Hu, Andreas Bulling, Sheng Li, Guoping Wang

Dataset PDF PPT Code Experimental Scenes Supplemental Material


Human visual attention in immersive virtual reality (VR) is key for many important applications, such as content design, gaze-contingent rendering, or gaze-based interaction. However, prior works typically focused on free-viewing conditions that have limited relevance for practical applications. We first collect eye tracking data of 27 participants performing a visual search task in four immersive VR environments. Based on this dataset, we provide a comprehensive analysis of the collected data and reveal correlations between users' eye fixations and other factors, i.e. users' historical gaze positions, task-related objects, saliency information of the VR content, and users' head rotation velocities. Based on this analysis, we propose FixationNet -- a novel learning-based model to forecast users' eye fixations in the near future in VR. We evaluate the performance of our model for free-viewing and task-oriented settings and show that it outperforms the state of the art by a large margin of 19.8% (from a mean error of 2.93° to 2.35°) in free-viewing and of 15.1% (from 2.05° to 1.74°) in task-oriented situations. As such, our work provides new insights into task-oriented attention in virtual environments and guides future work on this important topic in VR research.

Paper Video

Presentation Video

Related Work

Our related work:

EHTask: Recognizing User Tasks from Eye and Head Movements in Immersive Virtual Reality

Research progress of user task prediction and algorithm analysis (in Chinese)

Eye Fixation Forecasting in Task-Oriented Virtual Reality

Gaze Analysis and Prediction in Virtual Reality

DGaze: CNN-Based Gaze Prediction in Dynamic Scenes

Temporal Continuity of Visual Attention for Future Gaze Prediction in Immersive Virtual Reality

SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction


@article{hu21_FixationNet, title={FixationNet: Forecasting eye fixations in task-oriented virtual environments}, author={Hu, Zhiming and Bulling, Andreas and Li, Sheng and Wang, Guoping}, journal={IEEE Transactions on Visualization and Computer Graphics}, volume={27}, number={5}, pages={2681--2690}, year={2021}, publisher={IEEE} }