Exploring Data by Ear and Eye

Researchers of the St. Pölten University of Applied Sciences have published a state-of-the-art report on integrated audiovisual data analysis within the framework of the project SoniVis. For this purpose, the SoniVis team reviewed scientific works connecting seeing and hearing in data analysis. The research team will present the publication at the EuroVis in Denmark in May 2024 – the most prominent conference on data visualisation in Europe.

Sonification and visualisation

Researchers dedicated either to the visualisation or the sonification of data pursue a common goal: They want to make data interpretable for people. While the one group uses visual representations of data, the other group relies on auditive representations. Despite the overlaps between the two fields in terms of content, the respective research communities evolved largely in parallel over the past decades. In the project “SoniVis”, UAS researchers Kajetan Enge and Alexander Rind – under the leadership of UAS professors Wolfgang Aigner and Michael Iber – are working on a design theory in which visual and auditive channels complement each other. To this end, they examined scientific papers that already merge both techniques into a form of audiovisual representation. A new state-of-the-art report will be published soon that offers a current overview of audiovisual designs of data exploration and presentation.

“With this report on the state of the art in technology, we discuss a collection of works that bridge the gap between the two communities. In other words, a collection of works with the goal of combining the two techniques into a form of audiovisual representation that is more than the sum of its parts”, says Alexander Rind, researcher and project team members at the St. Pölten UAS’ Institute of Creative\Media/Technologies.

New Classification of Audiovisual Representations

Together with an international team of authors, Kajetan Enge and Alexander Rind reviewed 57 academic publications that were issued between 2011 and 2023. They categorised these by, e.g., data type, the supported analysis questions, and the research method used. This systematic categorisation made it possible to carry out a meta-analysis of the overlapping disciplines for the first time:

“In our data, we can see, for example, that traditional line charts are often combined with the widespread sonification technique of ‘parameter mapping sonification’”, says Kajetan Enge. “In parameter mapping sonification, sound qualities such as pitch or volume are assigned to certain data attributes in order to learn something about the data from the sound.”

Integrated Design

The reviewed publications encompass a diverse field of combined designs for data visualisation and sonification. At the same time, however, there is an imbalance between the visualised and sonified information.

“This phenomenon points to the fact that many designs use visualisation to offer an overview of the data, while sonification is applied to the analysis of details”, explains Kajetan Enge. “Researchers of both fields can use the results of the state-of-the-art report to understand the potentials and challenges of these integrated designs. It would make us very happy if our publication could inspire experts from both realms to work together.”

International Research 

International networking between researchers from sonification and visualisation is a key focus of the SoniVis project. For the state-of-the-art report, the research team cooperated with fellow experts from Sweden (Linköping University), Hong Kong (City University), Spain (Universidad de Deusto), and the USA (University of Utah) as well as the University of Music and Performing Arts Graz. In May 2024, the research team will present their publication at the EuroVis conference, the most important conference in the field of data visualisation in Europe, which is taking place in Denmark this year. In June, the report will be published in the journal “Computer Graphics Forum”.

About the Project 

“SoniVis” is a research project of the Institute of Creative\Media/Technologies at the St. Pölten UAS and is funded by the Austrian Science Fund (FWF). 

• You can read the publication here.
• Further information is available on the research website.