Research Areas


PLAY-IT: Realistic Immersion with Virtual Humans

The project “Realistic Immersion with Virtual Humans” has started on 01 January 2017 with a funding amount of SGD 499,968 by the National Research Foundation in Singapore. This project is in collaboration with Institute of Software, Chinese Academy of Sciences in Beijing, China. The Chinese team will lead Project 1 with Prof Wencheng Wang as lead Principal Investigator and the Singapore team will head Project 2 with Prof Nadia Magnenat Thalmann as lead Principal Investigator. The budget received by the Chinese team is RMB 1,998,200 (approximately SGD 404,656). The project duration is three years.

In this project, we introduce a new model of perception-attention-action process for virtual humans in order to improve the naturalness of their behaviour. The model of perception-attention-action includes:

  • Perception allows virtual humans to access knowledge of states of real users and other virtual humans (for example position, gesture and emotion) and information of both real and virtual environment.

  • Attention models the cognitive process of real human to focus on selected information with importance or interest. The “attention” of virtual humans in existing applications is mostly predefined. With our proposed model, virtual humans could intelligently and dynamically distribute their attention based on personality, emotion and events in virtual scenes.

  • Action is the procedure of decision-making and motion synthesis for virtual humans. We focus on how virtual humans acts in a mixed social scenario with both real and virtual humans.

To implement such a model, we will develop novel techniques, including the techniques for efficient perception of important visual information (like gestures and emotion), efficient attention of salient visual features to avatars, collaborative determination based on social communication, realistic activities of avatars, efficient refreshing of the virtual world. The developed novel techniques will be integrated into a prototype system to promote behaviour of virtual humans, and will be applied to some concrete applications, such as volleyball games in VR environments.

  • N. Jain, A. Wydra, W. Hai, N. Magnenat-Thalmann and D. Thalmann, Time scaled interactive object driven Multi-party VR, The Visual Computer, DOI: 10.1007/s00371-018-1539-1, Vol 34, Issue 6-8, pp. 887-897, May 04, 2018 (2nd Best Paper Award of CGI Conference 2018)