Pixel power

Imagine opening a door and walking into a strange universe where Newton’s laws do not apply and imagination rules the day. This can be achieved thanks to digital tools such as virtual reality (VR) and augmented reality (AR), which allow us to visualise and interact with digital information as if they were real.

At the Multi-plAtform Game Innovation Centre (MAGIC) at NTU’s School of Computer Science and Engineering (SCSE), we conduct research in VR such as 3D modelling, simulation and real-time graphics, as well as advance possibilities in AR, from computer vision to image processing. Beyond VR and AR, our research includes artificial intelligence and cloud computing.

Simulating deformable models
Avatars that resemble people in appearance, character and behaviour in crowds are important elements in any virtual environment, as well as in applications such as architectural validation and emergency evacuation. To develop more realistic avatar animations, the MAGIC team is working with other researchers from SCSE to simulate real-world stiffness and predict physical or mechanical behaviours under deformation.

Introducing the concept of fibre into deformable models, we use fibre materials to animate a skeletal character that has its skin modelled by a surface mesh. Unlike geometrically based skinning techniques that suffer from obvious volume distortion artefacts, our technique allows us to achieve real-time performance with large deformations (Figure 1).

Figure 1: Skeletal animation of a 3D model. (a) A surface mesh with underlying skeletal structure. (b) Proper skinning without volume distortion artefacts. Credit: Cai Jianping.

Relieving persistent neck pain
In a collaboration with Singapore’s Sengkang Health hospital, we are studying the neck mobility of patients who suffer from neck pain. By wearing a VR headset while following the path of an object, patients generate data that enable doctors to analyse their cervical joint position sense as well as the accuracy and time taken to complete the task (Figure 2).

Figure 2: Real-time head motion measurement with a mobile VR headset. Credit: MAGIC.

Our VR approach makes it possible for patients to complete the assessment remotely at home. Fun and interactive tasks can also be incorporated to increase compliance with exercises. On a population scale, we aim to create a registry and analyse the collected data.

A blast to the past
VR and AR are also excellent tools to make history accessible to the general public. In collaboration with the National Archives of Singapore and NTU’s Wee Kim Wee School of Communication and Information, our team developed a mobile AR-based game that features scenes from Bukit Merah, a township in the heart of Singapore that includes the country’s first housing estate (Figure 3).

Figure 3: A player “collecting” a bag of rice. Credit: MAGIC.

As players “visit” historical locations, we can track the paths they take and study their behaviour as individuals or as a group. We can also push information to them during their expedition. Other applications for our AR-based platform include supporting group interaction and learning.

Using drones in crisis management training
In collaboration with ST Engineering and NTU’s Renaissance Engineering Programme (a dual-degree programme that blends engineering with business and humanities), we are developing an interactive AR drone platform that can receive live video feeds from drones and overlay virtual objects onto the real world (Figure 4).

Figure 4: AR drone platform, with a flying drone next to the whiteboard stand. The drone’s camera view—overlaid with virtual objects—is shown in the inset, bottom left. Credit: Jillian Heng, Lyanne Loh and Oo Guolin (Renaissance Engineering Programme).

To provide a seamless and believable experience, we are using a class of deep learning technologies called convolutional neural networks to recover the camera’s position and orientation accurately and in real time, in combination with pre-trained data. One application for the platform is in crisis management training, as such situations can be challenging to stage with real actors.

Launching future VR/AR enthusiasts
To facilitate learning among students and researchers, NTU has set up an Interactive Digital Centre (IDC) in collaboration with multinational VR and AR software developer EON Reality. IDC offers a cutting-edge virtual showroom as well as a lab and training facility, providing a launching pad for future generations of VR and AR professionals and enthusiasts.

Author: Seah Hock Soon
Prof Seah Hock Soon is Director of the Multi-plAtform Game Innovation Centre (MAGIC) at NTU’s School of Computer Science and Engineering. An expert in computer graphics and imaging research, he invented a drawing and animation software called Computer-Assisted Cel Animation (CACANi), and a 3D object recognition and scene recovery technique used in AR systems called “Structure from Simulated Motion”.
Watch a video of Prof Seah’s research project “Direct 3D stylisation pipelines” in SIGGRAPH Real Time Live (2017), or read about his research in Graphical Simulation of Deformable Models (2016), Springer International Publishing, Switzerland, DOI: 10.1007/978-3-319-51031-6_6.
The article appeared first in NTU’s research & innovation magazine Pushing Frontiers (issue #13, June 2018).

You may also like...