BeingTogether Centre 
BeingTogether Centre

BeingTogether Centre has started on 01 October 2016 with a funding amount of SGD 6,000,000 by the National Research Foundation in Singapore. The project duration is three years. The host institution is the Nanyang Technological University (NTU) and the lead Principal Investigator is Prof Nadia Magnenat Thalmann. This centre is in partnership with the University of North Carolina (UNC) at Chapel Hill.

The goal of this centre is to work on real and virtual humans, social robots, real and virtual objects that can simultaneously interact in a space that is a geometrically consistent blend of different remote real spaces and synthetic environments. BeingTogether means (1) occupying the same space; and (2) everybody is aware of everyone and can interact with each of them. To achieve this goal, there are two key issues: the building of this blended space, and the mixed society, which inhibits this space.


Project 1: A Fusion of Worlds – Immersive Telepresence

This project will achieve a geometrically consistent blend of different remote real spaces and synthetic environments by addressing three kinds of challenges:

(1) Real-time high fidelity scene acquisition;

(2) Practical, unencumbering wearable displays; and

(3) Immersive fusion across sites.

Project 2: Mixed Society of People, Virtual Humans and Social Robots in TelePresence

This project will build low cost robots and achieve the smooth integration of a mixed society in the blended space, with four main challenges:

(1) From any body and face, create the outer shelf of a humanoid robot through 3D virtual modelling and 3D printing;

(2) Each partner (real, virtual, telepresent, robot, autonomous or not) should be characterized by affects (including personality and emotion), memory, and interactions with gestures, facial expressions and dialogue;

(3) Each partner should be aware of all other partners and the envionrment, and able to collaborate and interact, and;

(4) Each partner (except virtual humans) should be able to interact physically with the environment, especially reaching, grasping and exchanging objects.

The two projects are completed with concrete two scenarios including the technical innovations of both projects. We will provide a first demonstrator with Humanoid Nadine robot being a real receptionist and being able to communicate with other people elsewhere and a second demonstrator where children will enjoy the companionship of a social robot to learn and discover new things together. The social robot will also be able to monitor the children as well as communicate directly with each parents’ child. The parents will also be able to guide the robot distantly whenever they like to do it and interfere with the children.

  • T. Deng, J. Cai, T. J. Cham and J. Zheng, Multiple consumer-grade depth camera registration using everyday objects,, Image and Vision Computing, Vol. 62, Pp. 1-7, June 2017

  • C. Y. Wong and G. Seet, Workload, awareness and automation in multiple-robot supervision, International Journal of Advanced Robotic Systems, Vol. 14, Issue 3, DOI: 10.1177/1729881417710463, June 2017

  • P. K. Jayaraman, C. W. Fu, J. Zheng, X. Liu and T. T. Wong, Globally Consistent Wrinkle-Aware Shading of Line Drawings, IEEE Transactions on Visualization and Computer Graphics, Vol. PP, Issue 99, pp. 01-01, DOI: 10.1109/TVCG.2017.2705182, May 2017

  • D. Thalmann, Sensors and Actuators for HCI and VR: A Few Case Studies, Frontiers in Electronic Technologies, Springer, pp. 65-83, April 2017

  • N. Magnenat Thalmann, L. Tian and F. Yao, Nadine: A Social Robot that Can Localize Objects and Grasp Them in a Human Way, Frontiers in Electronic Technologies, Springer, pp. 1-23, April 2017

  • Y. Cai, R. Chiew, Z.T. Nay, C. Indhumathi, and L. Huang, Design and development of VR learning environments for children with ASD, Interactive Learning Environments , Vol. 25, Issue 3, pp. 01-12, DOI: 10.1080/10494820.2017.1282877, March 2017

  • D. Xu, Q. Duan, J. Zheng, J. Zhang, J. Cai and T.J. Cham, Shading-based Surface Detail Recovery under General Unknown Illumination, IEEE Transactions on Pattern Analysis and Machine Intelligence , Vol. PP, Issue 99, pp. 01-01, DOI: 10.1109/TPAMI.2017.2671458 February 2017