Zhiming Hu

Stuttgart Center for Simulation Science (SimTech), University of Stuttgart

zhiming.hu (at) simtech.uni-stuttgart.de Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building Google Scholar, Github, Curriculum Vitae, Essays

Biography

Zhiming Hu is a post-doctoral researcher in the Perceptual User Interfaces Group and Computational Biophysics and Biorobotics Group, at the University of Stuttgart, Germany since August 2022. He obtained his Ph.D. degree in Computer Software and Theory from Peking University, China in 2022. He received his Bachelor's degree in Optical Engineering from Beijing Institute of Technology, China in 2017. His research interests include virtual reality, visual attention, human-computer interaction, eye tracking, and human intention prediction. He served as a reviewer for IEEE VR, ISMAR, TVCG, IMWUT, IJHCI, CVPR, ECCV, and ICCV.

Honors, Awards, and Scholarships

• Outstanding Graduates Prize in Peking University, 2022
• National Scholarship, 2021
• TVCG Best Journal Nominees Award (IEEE VR 2021)
• CSC (China Scholarship Council) Scholarship, 2020
• Second-Class Scholarship of Peking University, 2020
• Chancellor's Scholarship, 2020
• Leo KoGuan Scholarship, 2019
• Leader Scholarship, 2017
• Outstanding Graduates Prize in Beijing Institute of Technology, 2017
• National Scholarship, 2016
• Honorable Mention of Mathematical Contest in Modeling, 2016
• National Encouragement Scholarship, 2015
• Third Prize of the National College Students Composition Competition, 2015
• Third Prize of Beijing Humanistic Knowledge Competition, 2014
• National Scholarship, 2014

Publications

1. 

   Intentional Head-Motion Assisted Locomotion for Reducing Cybersickness

   Zehui Lin, Xiang Gu, Sheng Li, Zhiming Hu, Guoping Wang

   IEEE Transactions on Visualization and Computer Graphics, 2022.

   Abstract Links BibTeX

   We present an efficient locomotion technique that can reduce cybersickness through aligning the visual and vestibular induced self-motion illusion. Our locomotion technique stimulates proprioception consistent with the visual sense by intentional head motion, which includes both the head's translational movement and yaw rotation. A locomotion event is triggered by the hand-held controller together with an intended physical head motion simultaneously. Based on our method, we further explore the connections between the level of cybersickness and the velocity of self motion through a series of experiments. We first conduct Experiment 1 to investigate the cybersickness induced by different translation velocities using our method and then conduct Experiment 2 to investigate the cybersickness induced by different angular velocities. Our user studies from these two experiments reveal a new finding on the correlation between translation/angular velocities and the level of cybersickness. The cybersickness is greatest at the lowest velocity using our method, and the statistical analysis also indicates a possible U-shaped relation between the translation/angular velocity and cybersickness degree. Finally, we conduct Experiment 3 to evaluate the performances of our method and other commonly-used locomotion approaches, i.e., joystick-based steering and teleportation. The results show that our method can significantly reduce cybersickness compared with the joystick-based steering and obtain a higher presence compared with the teleportation. These advantages demonstrate that our method can be an optional locomotion solution for immersive VR applications using commercially available HMD suites only.

   Paper: lin22_Intentional.pdf

   @article{lin22_Intentional, title = {Intentional Head-Motion Assisted Locomotion for Reducing Cybersickness}, author = {Lin, Zehui and Gu, Xiang and Li, Sheng and Hu, Zhiming and Wang, Guoping}, year = {2022}, journal = {IEEE Transactions on Visualization and Computer Graphics} }

1. 

   EHTask: Recognizing User Tasks from Eye and Head Movements in Immersive Virtual Reality

   Zhiming Hu, Andreas Bulling, Sheng Li, Guoping Wang

   IEEE Transactions on Visualization and Computer Graphics, 2022.

   Abstract Links BibTeX Project

   Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e. Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask -- a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% vs. 62.8%) and on a real-world dataset (61.9% vs. 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.

   doi: 10.1109/TVCG.2021.3138902

   Paper: hu22_EHTask.pdf

   @article{hu22_EHTask, title = {EHTask: Recognizing User Tasks from Eye and Head Movements in Immersive Virtual Reality}, author = {Hu, Zhiming and Bulling, Andreas and Li, Sheng and Wang, Guoping}, year = {2022}, journal = {IEEE Transactions on Visualization and Computer Graphics} }

1. 

   Research progress of user task prediction and algorithm analysis (in Chinese)

   Zhiming Hu, Sheng Li, Meng Gai

   Journal of Graphics, 2021, 42(3): 367-375.

   Abstract Links BibTeX Project

   Users’ cognitive behaviors are dramatically influenced by the specific tasks assigned to them. Information on users’ tasks can be applied to many areas, such as human behavior analysis and intelligent human-computer interfaces. It can be used as the input of intelligent systems and enable the systems to automatically adjust their functions according to different tasks. User task prediction refers to the prediction of users’ tasks at hand based on the characteristics of his or her eye movements, the characteristics of scene content, and other related information. User task prediction is a popular research topic in vision research, and researchers have proposed many successful task prediction algorithms. However, the algorithms proposed in prior works mainly focus on a particular scene, and comparison and analysis are absent for these algorithms. This paper presented a review of prior works on task prediction in scenes of images, videos, and real world, and detailed existing task prediction algorithms. Based on a real-world task dataset, this paper evaluated the performances of existing algorithms and conducted the corresponding analysis and discussion. As such, this work can provide meaningful insights for future works on this important topic.

   doi: http://www.txxb.com.cn/CN/10.11996/JG.j.2095-302X.2021030367

   Paper: hu21_UserTask.pdf

   @article{hu21_User, title = {Research progress of user task prediction and algorithm analysis (in Chinese)}, author = {Hu, Zhiming and Li, Sheng and Gai, Meng}, year = {2021}, journal={Journal of Graphics}, doi = {http://www.txxb.com.cn/CN/10.11996/JG.j.2095-302X.2021030367}, volume = {42}, number = {3}, pages = {367-375} }

1. 

   Eye Fixation Forecasting in Task-Oriented Virtual Reality

   Zhiming Hu

   Proceedings of the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, 2021: 707-708.

   Abstract Links BibTeX Project

   In immersive virtual reality (VR), users' visual attention is crucial for many important applications, including VR content design, gaze-based interaction, and gaze-contingent rendering. Especially, information on users' future eye fixations is key for intelligent user interfaces and has significant relevance for many areas, such as visual attention enhancement, dynamic event triggering, and human-computer interaction. However, previous works typically focused on free-viewing conditions and paid less attention to task-oriented attention. This paper aims at forecasting users' eye fixations in task-oriented virtual reality. To this end, a VR eye tracking dataset that corresponds to different users performing a visual search task in immersive virtual environments is built. A comprehensive analysis of users' eye fixations is performed based on the collected data. The analysis reveals that eye fixations are correlated with users' historical gaze positions, task-related objects, saliency information of the VR content, and head rotation velocities. Based on this analysis, a novel learning-based model is proposed to forecast users' eye fixations in the near future in immersive virtual environments.

   doi: 10.1109/VRW52623.2021.00236

   Paper: hu21_Eye.pdf

   @inproceedings{hu21_Eye, author={Hu, Zhiming}, title = {Eye Fixation Forecasting in Task-Oriented Virtual Reality}, booktitle={Proceedings of the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops}, year = {2021}, pages={707-708}, organization={IEEE} }

1. 

   FixationNet: Forecasting Eye Fixations in Task-Oriented Virtual Environments

   Zhiming Hu, Andreas Bulling, Sheng Li, Guoping Wang

   IEEE Transactions on Visualization and Computer Graphics, 2021, 27(5): 2681-2690.

   Abstract Links BibTeX Project TVCG Best Journal Nominees Award

   Human visual attention in immersive virtual reality (VR) is key for many important applications, such as content design, gaze-contingent rendering, or gaze-based interaction. However, prior works typically focused on free-viewing conditions that have limited relevance for practical applications. We first collect eye tracking data of 27 participants performing a visual search task in four immersive VR environments. Based on this dataset, we provide a comprehensive analysis of the collected data and reveal correlations between users' eye fixations and other factors, i.e. users' historical gaze positions, task-related objects, saliency information of the VR content, and users' head rotation velocities. Based on this analysis, we propose FixationNet -- a novel learning-based model to forecast users' eye fixations in the near future in VR. We evaluate the performance of our model for free-viewing and task-oriented settings and show that it outperforms the state of the art by a large margin of 19.8% (from a mean error of 2.93° to 2.35°) in free-viewing and of 15.1% (from 2.05° to 1.74°) in task-oriented situations. As such, our work provides new insights into task-oriented attention in virtual environments and guides future work on this important topic in VR research.

   doi: 10.1109/TVCG.2021.3067779

   Paper: hu21_FixationNet.pdf

   @article{hu21_FixationNet, title={FixationNet: Forecasting eye fixations in task-oriented virtual environments}, author={Hu, Zhiming and Bulling, Andreas and Li, Sheng and Wang, Guoping}, journal={IEEE Transactions on Visualization and Computer Graphics}, volume={27}, number={5}, pages={2681--2690}, year={2021}, publisher={IEEE} }

1. 

   Gaze Analysis and Prediction in Virtual Reality

   Zhiming Hu

   Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, 2020: 543-544.

   Abstract Links BibTeX Project

   In virtual reality (VR) systems, users’ gaze information has gained importance in recent years. It can be applied to many aspects, including VR content design, eye-movement based interaction, gaze-contingent rendering, etc. In this context, it becomes increasingly important to understand users’ gaze behaviors in virtual reality and to predict users’ gaze positions. This paper presents research in gaze behavior analysis and gaze position prediction in virtual reality. Specifically, this paper focuses on static virtual scenes and dynamic virtual scenes under free-viewing conditions. Users’ gaze data in virtual scenes are collected and statistical analysis is performed on the recorded data. The analysis reveals that users’ gaze positions are correlated with their head rotation velocities and the salient regions of the content. In dynamic scenes, users’ gaze positions also have strong correlations with the positions of dynamic objects. A data-driven eye-head coordination model is proposed for realtime gaze prediction in static scenes and a CNN-based model is derived for predicting gaze positions in dynamic scenes.

   doi: 10.1109/VRW50115.2020.00123

   Paper: hu20_Gaze.pdf

   @inproceedings{hu20_Gaze, author={Hu, Zhiming}, title = {Gaze Analysis and Prediction in Virtual Reality}, booktitle={Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops}, year = {2020}, pages={543--544}, organization={IEEE} }

1. 

   DGaze: CNN-Based Gaze Prediction in Dynamic Scenes

   Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, Dinesh Manocha

   IEEE Transactions on Visualization and Computer Graphics, 2020, 26(5): 1902-1911.

   Abstract Links BibTeX Project

   We conduct novel analyses of users' gaze behaviors in dynamic virtual scenes and, based on our analyses, we present a novel CNN-based model called DGaze for gaze prediction in HMD-based applications. We first collect 43 users' eye tracking data in 5 dynamic scenes under free-viewing conditions. Next, we perform statistical analysis of our data and observe that dynamic object positions, head rotation velocities, and salient regions are correlated with users' gaze positions. Based on our analysis, we present a CNN-based model (DGaze) that combines object position sequence, head velocity sequence, and saliency features to predict users' gaze positions. Our model can be applied to predict not only realtime gaze positions but also gaze positions in the near future and can achieve better performance than prior method. In terms of realtime prediction, DGaze achieves a 22.0% improvement over prior method in dynamic scenes and obtains an improvement of 9.5% in static scenes, based on using the angular distance as the evaluation metric. We also propose a variant of our model called DGaze_ET that can be used to predict future gaze positions with higher precision by combining accurate past gaze data gathered using an eye tracker. We further analyze our CNN architecture and verify the effectiveness of each component in our model. We apply DGaze to gaze-contingent rendering and a game, and also present the evaluation results from a user study.

   doi: 10.1109/TVCG.2020.2973473

   Paper: hu20_DGaze.pdf

   @article{hu20_DGaze, title={DGaze: CNN-Based Gaze Prediction in Dynamic Scenes}, author={Hu, Zhiming and Li, Sheng and Zhang, Congyi and Yi, Kangrui and Wang, Guoping and Manocha, Dinesh}, journal={IEEE Transactions on Visualization and Computer Graphics}, volume={26}, number={5}, pages={1902--1911}, year={2020}, publisher={IEEE} }

1. 

   Temporal continuity of visual attention for future gaze prediction in immersive virtual reality

   Zhiming Hu, Sheng Li, Meng Gai

   Virtual Reality & Intelligent Hardware, 2020, 2(2): 142-152.

   Abstract Links BibTeX Project

   Background Eye tracking technology is receiving increased attention in the field of virtual reality. Specifically, future gaze prediction is crucial in pre-computation for many applications such as gaze-contingent rendering, advertisement placement, and content-based design. To explore future gaze prediction, it is necessary to analyze the temporal continuity of visual attention in immersive virtual reality. Methods In this paper, the concept of temporal continuity of visual attention is presented. Subsequently, an autocorrelation function method is proposed to evaluate the temporal continuity. Thereafter, the temporal continuity is analyzed in both free-viewing and task-oriented conditions. Results Specifically, in free-viewing conditions, the analysis of a free-viewing gaze dataset indicates that the temporal continuity performs well only within a short time interval. A task-oriented game scene condition was created and conducted to collect users' gaze data. An analysis of the collected gaze data finds the temporal continuity has a similar performance with that of the free-viewing conditions. Temporal continuity can be applied to future gaze prediction and if it is good, users' current gaze positions can be directly utilized to predict their gaze positions in the future. Conclusions The current gaze's future prediction performances are further evaluated in both free-viewing and task-oriented conditions and discover that the current gaze can be efficiently applied to the task of short-term future gaze prediction. The task of long-term gaze prediction still remains to be explored.

   doi: 10.1016/j.vrih.2020.01.002

   Paper: hu20_Temporal.pdf

   @article{hu20_Temporal, title={Temporal continuity of visual attention for future gaze prediction in immersive virtual reality}, author={Hu, Zhiming and Li, Sheng and Gai, Meng}, journal={Virtual Reality and Intelligent Hardware}, volume={2}, number={2}, pages={142--152}, year={2020}, publisher={Elsevier} }

1. 

   SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction

   Zhiming Hu, Congyi Zhang, Sheng Li, Guoping Wang, Dinesh Manocha

   IEEE Transactions on Visualization and Computer Graphics, 2019, 25(5): 2002-2010.

   Abstract Links BibTeX Project

   We present a novel, data-driven eye-head coordination model that can be used for realtime gaze prediction for immersive HMD-based applications without any external hardware or eye tracker. Our model (SGaze) is computed by generating a large dataset that corresponds to different users navigating in virtual worlds with different lighting conditions. We perform statistical analysis on the recorded data and observe a linear correlation between gaze positions and head rotation angular velocities. We also find that there exists a latency between eye movements and head movements. SGaze can work as a software-based realtime gaze predictor and we formulate a time related function between head movement and eye movement and use that for realtime gaze position prediction. We demonstrate the benefits of SGaze for gaze-contingent rendering and evaluate the results with a user study.

   doi: 10.1109/TVCG.2019.2899187

   Paper: hu19_SGaze.pdf

   @article{hu19_SGaze, title={SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction}, author={Hu, Zhiming and Zhang, Congyi and Li, Sheng and Wang, Guoping and Manocha, Dinesh}, journal={IEEE Transactions on Visualization and Computer Graphics}, volume={25}, number={5}, pages={2002--2010}, year={2019}, publisher={IEEE} }