Immersing into Artificial Media Worlds

The St. Pölten UAS‘ “Immersive Media Lab“ explores new forms of storytelling through Augmented and Virtual Reality (AR/VR) and focuses on immersion by diving into virtual worlds.

In the laboratory, students and researchers develop projects for areas such as media art, Industry 4.0, and digital health information. The results shall benefit the industry, the health sector and the creative industries.

Storytelling in Art, Science and the Industry

Since its foundation two years ago, the laboratory has been dedicated to five concrete areas of application, so-called use cases: an informative industrial hearing protection, information for patients, digital art formats for 3D motion capture, and concepts for communication in the Industrial Internet of Things as well a potential analysis for fictional VR experience.

“Storytelling, i.e. conveying contents through stories, has always been a natural thing to do in the arts. Now, it has been gaining increasing importance in science and the economy. The technologies of Augmented and Virtual Reality not only constitute an interface between industrial application and artistic research, they also offer the opportunity to rethink the areas storytelling can be used in”, says Franziska Bruckner, head of the project and the research group Media Creation at the St. Pölten UAS.

The five use cases develop immersive media technology for media art, the creative industries, the automotive industry, Industry 4.0, and the digital health sector. Here is an overview of the laboratory’s projects:

Combining Art and Sciences

The project “Intermedia Motion Tracking in AR/VR“ uses motion capturing and motion tracking techniques to record a dancer’s movements and translates them into pictures and films for social media. Choreographed body movements, gestures, motion sequences and choreographic patterns are defined and digitalised to analyse them as storytelling motion paths.

The project combines modern media technology, art and science by examining the relationship between body and space as well as body posture, elasticity, speed and rhythm of the choreography and its application in artistic and experimental immersive storytelling scenarios and media projects.  

Goose Bumps: Measuring Immersion

The use case “Immersive Video Interaction“ analyses fictional Virtual Reality short films from the entertainment industry with regard to their narrative models, interaction concepts and degree of immersion. To measure the subjective experiences of immersing into a virtual world, viewers are asked to describe their feeling of being present during their virtual experience.  

Moreover, the project team is going to apply criteria that can be measured objectively: by means of electrical skin resistance, heart frequency, electroencephalograms, and electrical activities of facial muscles, the researchers seek to determine the impact the films have on the viewer. The user tests, which had to be postponed due to the COVID-19 pandemic, are expected to take place towards the end of this year or next year under the strictest hygiene safety measures.

Alleviation of Preoperative Anxiety: Enlightening Patients with Augmented Reality

The use case “Enlightening Patients with Augmented Reality (EPAR)” tests the potential of Augmented Reality to optimise patient information. For this purpose, a mobile app on squint surgery was developed and tested. At their own pace, patients interested in a surgery may obtain additional information, which doctors often lack time to provide.

Informative Hearing Protection for Industry Workers

In the use case “Auditory Augmented Reality in Production (AARiP)”, hearing protections, which are mandatory in many production facilities, are developed further by researchers and students of the St. Pölten UAS and turned into an acoustic information interface and an assistance system. Processed machine and surrounding noises are represented acoustically in (quasi) real time to optimise the perception of process-relevant deviations from the norm.

The system provides machine- and production-related data and key figures, which are relevant for production and have been processed for condition monitoring. In addition, the hearing protection shall serve as headphones for employees for verbal (wireless) communication. In this context, (data) storytelling can be understood as the situation-specific processing and display of sounds and data with a high information content for skilled workers.

Visual Data for Malfunctions in Machines

The most recent use case “Augmented Reality for the Industrial Internet of Things“ has only started this year. In partnership with the Lower Austrian company Neumann Aluminium, the project team researches the use of Augmented Reality technologies in modern, industrial production environments to support employees in work and maintenance processes. 

Through a Microsoft Hololens, employees may directly and immediately access machine data in the event of malfunctions, in cases where factories normally do not provide displays. In addition, the Hololens keeps their hands free for working.

Recently, the Immersive Media Lab Council convened online to discuss the current state of the projects.

Extensive Communication Activities

The Immersive Media Lab Team have seized a plethora of opportunities to present their work and results to the specialist community and the wider public and have published a number of articles in specialist magazines and for conferences.

The project team members have presented their work at conferences such as the following: the Forschungstag NÖ (Lower Austrian Day of Research), the Long Night of Research, the Audiomesse (audio fair), the IEEEVR Conference (Conference on Virtual Reality and 3D User Interfaces), the EX\\Immersio Symposium, and the UAS conferences XChange Reality and IConCMT.

Immersive Media Lab

The Immersive Media Lab is funded by the Federal Ministry for Digital and Economic Affairs within the framework of the programme COIN “Aufbau” by the Austrian Research Promotion Agency (FFG).