Breaking the Wall

Playful interfaces for audience participation and artistic expression in musical live performances

ART-BASED RESEARCH (FWF PEEK)

With Breaking The Wall, we explore how to use technology to involve the audience in music performances. We design and develop different interactive systems together with artists. These systems will be deployed in a public performance in spring 2017 in the Kuppelsaal of the Vienna University of Technology. 

The main research question we want to answer is:

“Which new ways of artistic expression emerge in a popular form of music performance when using playful interfaces for audience participation to facilitate interactivity among everybody involved?”

The results of the project will be situated at the interdisciplinary intersection of art, music and technology. We will present structured and evaluated insights into the unique relation between performers and audience, leading to tested and documented new artistic ways of musical expression that future performances can build on. We will further deliver a tool-set with new interfaces and collaborative digital instruments. The results of the project will be highly relevant to musical practice, and contribute to theory from the areas of media arts and musicology.

News

Breaking The Wall - Ars Electronica Performance

Thursday, 7. September 2017, Post City, Linz.

Performance null.head. Ars Electronica Festival - Opening Concerts. Schedule/Line-up

Breaking The Wall - Interactive Sound Performance - The Event

Friday June 2nd, 7:30 PM, Kuppelsaal, TU Vienna, Karlsplatz 13

See the archived event website http://breakingthewall.at which is not online anymore as screenshot or PDF.

See video on top or on youtube.

Project Work

This section reports about ongoing work and prototypes created together with musicians.

DJ-mediated Distributed Smartphone performance

Together with the composer, DJ and musician Electric Indigo (Susanne Kirchmayr) we develop a performance with the goal to break up the traditional relation between artist and audience by making the audience an extension of the artistic intent and expression of the performance. By using an app developed during the course of the project (Poème Numérique by Christoph Bartmann) the smartphones of the attendees are transformed into mobile sound sources. On the one hand these can be manipulated in real-time by the composer and musician, on the other they are freed of a static location-dependance by the behaviour and movement of the audience. This way the audience (as a swarm) is able to independently affect the distribution of sound thereby acting as a dynamic mediator between performer and performance space. This bridge is realised technologically through the use of ultra sound (high-frequency sound IDs), which connects the audience's smartphones to the performer without the need to join a WiFi or using a mobile phone network, thus lowering the technological threshold for participation.

multi-sensory Audience experience

The collaboration with the performance artists Fuckhead (Didi Bruckmayr, Chris Bruckmayr et al.) focuses on the technological and dramaturgical connection of body, sound, light and room. Through this multi-sensory experience, the audience should be able to reflect on and question digital surveillance and technological authority as it may be part of technology-mediated audience participation. This kind of embodied and technological intervention creates an experimental situation where accepted customs, habits, and eerines convene interchangingly. Technically, the audience will be tracked in the room and becomes a characteristic parameter of the whole performance architecture.

3D Tracking of Multiple Balloons Using Computer Stereo Vision

The developed balloon tracking system focuses on interactive audience participation in live concerts using multiple balloons. The balloon tracking system is able to track 3D positions of several balloons during the concert in real time. Technically, the system uses two static cameras placed in front of the audience to record videos of the balloons from two different angles. In every frame of these video streams, all balloons are detected in 2D. After detection in 2D, the system calculates their 3D position. Once the system has calculated the positions, it sends the position of each balloon using OSC. The system is designed to be robust enough to work in different kinds of conditions that may occur during the live concert, such as fogginess and dynamic lighting. The system is made in C++ using the computer vision library OpenCV. The presented system was implemented by Karlo Dumbović as part of an internship in collaboration with the Computer Vision Lab.

border.games - Game-based audience participation

In the course of their diploma theses Naida Comaga & Lukas Gartlehner developed a prototype for audience participation with OpenCV, Processing and Arduino. By using a projection canvas that seperated the audience from the musicians, the wall that was to be broken was visualized in an interactive manner. During the concert in which the prototype was tested the movement of the audience was tracked by a camera and used as an input for both projection and sound. The more the audience jumped, danced and moved around, the sooner parts of the canvas fell to the ground, opening up the stage that was hidden behind the projection. The collaboration with musicians Lea Föger (Singer) and Melanie Asböck (Violin) as well as the support of the Kulturverein mo.ë made it possible to test this prototype in a unique concert setup.

Publications

Project-related publications, talks and theses as well as open source tools to download are listed below.

Performances, EXhibitions and other EVENTs

Performance Breaking the Wall. Kuppelsaal TU Wien, 2.6.2017 (Archived website, Facebook event)

Symposium Breaking the Wall. Ars Electronica Festival - Conference Track. Post City, Linz, 7.9.2017 (Event website)

Performance null.head. Ars Electronica Festival - Opening Concerts. Post City, Linz, 7.9.2017 (Schedule/Line-up)

Artistic Research Contemporary Code. Artistic research exhibition. City University of Hong Kong. 29.10.2015.

Conference Proceedings

Kayali, F. Hödl, O. Bartmann, C. Kühn, U. Wagensommerer, T. Mateus-Berr, R. 2018. Learnings from an Iterative Design Process for Technology-Mediated Audience Participation (TMAP) using Smartphones. In EAI Endorsed Transactions on Creative Technologies, 5(14), pp.1-7. (Download PDF)

Hödl, O. Fitzpatrick, G. Kayali, F. Holland, S. 2017. Design Implications for Technology-Mediated Audience Participation in Live Music. In Proceedings of Sound and Music Computing 2017, Helsinki, FI. pp.28-24. (Download PDF)

Kayali, F. Hödl, O. Fitzpatrick, G. Purgathofer, P. Filipp, A. Mateus-Berr, R. Kühn, U. Wagensommerer, T. Kretz, J. Kirchmayr, S. 2017. Playful Technology-Mediated Audience Participation in a Live Music Event. The ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (CHI PLAY) 2017, Amsterdam, NL, Oct 15th - 18th. pp. 437-443.

Reichl, P. Löw, C. Schröder, S. Schmidt, T. Schatzl, B. Lushaj, V. Hödl, O. Güldenpfennig, F. Widauer C. 2016. The Salome Experience: Opera Live Streaming and Beyond. ACM SIGCHI 2016, San Jose, USA. pp. 728-737. (Honorable Mention Award)

Güldenpfennig, F. Hödl, O. Reichl, P. Löw, C. Gartus, A. Pelowski, M. 2016. TASK: Introducing The Interactive Audience Sensor Kit. TEI 2016, Eindhoven, Netherlands. pp. 484-454.

Hödl, O., Kayali, F., Fitzpatrick, G. & Holland, S., 2016. TMAP Design Cards for Technology- Mediated Audience Participation in Live Music. In Workshop position paper for CHI 2016. pp. 1–4. (Download PDF)

Kayali, F., Bartmann, C., Hödl, O., Mateus-Berr, R., Pichlmair, M., 2016. Poème Numérique: Technology-Mediated Audience Participation (TMAP) using Smartphones and High-Frequency Sound IDs. In Proceedings of the INTETAIN 2016 8th International Conference on Intelligent Technologies for Interactive Entertainment, June 28-30 2016, Utrecht, NL. Springer, Cham.

BOOK CHAPTERS

Hödl, O. Kayali, F. Fitzpatrick, G. Holland, S. 2019. TMAP Design Cards for Technology-mediated Audience Participation in Live Music. In New Directions in Music and Human-Computer Interaction. S. Holland, T. Mudd, K. Wilkie, A. McPherson, M. Wanderley (eds.), Springer.

Poster PRESENTATIONS

Kayali, F. Hödl, O. Mateus-Berr, R. 2017. The Art-based Research Project Breaking The Wall, Poster presentation: Open House 2017, Zentrum Fokus Forschung, University of Applied Arts Vienna; 10-25-2017.

ThesEs

Comaga, N. Game-based audience participation in live performances (tentative title). Master Thesis, Vienna University of Technology. (ongoing)

Filipp, A., 2018. Evaluation of a music event with Technology-Mediated Audience Participation. Bachelor Thesis, Vienna University of Technology. (Download PDF)

Gartlehner, L., 2017. Game-based audience participation in live performances. Master Thesis, Vienna University of Technology. (Download PDF)

Hödl, O., 2016. The Design of Technology-Mediated Audience Participation in Live Music. PhD Thesis, Vienna University of Technology. (Download PDF)

Bartmann, C., 2016. Exploring audience participation in live music with a mobile application. Master Thesis, Vienna University of Technology. (Download PDF)

Pfeiffer, D., 2016. Simulation in Audience Participation for Live Music. Bachelor Thesis, Vienna University of Technology. (Download PDF)

Eigner, T., 2015. Investigating a color music interaction system in a collaborative music performance. Master Thesis, Vienna University of Technology. (Download PDF)

Technical Reports

Dumbovic, K., 2016. 3D Tracking of Multiple Balloons Using Computer Stereo Vision. Vienna University of Technology, Human Computer Interaction Group (E187/2) and Computer Vision Lab (E183/2). (Download PDF)

TALKS (INVITED)

Mateus-Berr, R. Delgado, J. Lerchbaumer, A. Kühn, U. Kayali, F. Hödl, O. Breaking the Wall for Critical Requests on Digital Evolution 4.0. Extended abstract and Talk: INTERVENTIONS: Scientific and Social Interventions in Art Education European InSEA Congress 2018, Aalto University, Helsinki, 18.-21.06.2018.

Hödl, O. Breaking The (Imaginary) Wall between Performers and their Audience in Live Music, All Around Audio Symposium. FH. St. Pölten, 29.11.2017.

Kayali, F. Breaking The Wall. Research Salon - Artificial Intelligence, University of Applied Arts Vienna, 12.11.2017.

Hödl, O. Music Computing from Opera to Avant-garde. Forum Medientechnik, All Around Audio Symposium. FH St. Pölten, 23.11.2016.

Hödl, O. Interaktive Publikumsbeteiligung in Live-Musik. Symposium PopNet Austria. University of Music and Performing Arts Vienna, 11.12.2015.

Kayali, F. Towards Bridging The Gap In A Musical Live Performance. Vienna Talk 2015 on Music Acoustics - "Bridging the Gaps". Wien, 16.09.2015 - 19.09.2015; in: Hödl, O. Kayali, F. Fitzpatrick, G. Holland, S. Proceedings of the Third Vienna Talk on Music Acoustics, (2015), S. 221.

EVENT COLLABORATION AND CO-ORGANISATION

3. Waves Music Hackday. WUK, Wien 29. & 30.10.2017

2. Waves Music Hackday. WUK, Wien 01.10.2016

1. Waves Music Hackday. Alte Post, Wien 03.10.2015

Open source Tools

The technologies (e.g. software) created during the project for the three performances are available open source. If you have any questions in using these technologies, please contact the project team. (See contact details at the bottom of this page.)

Download material for performance 1: “iStressTest”

Download material for performance 2: “Poème Numérique”

Download material for performance 3: “The Polluting Noise of Senseless Things”

People and Institutions

Breaking The Wall is funded by FWF PEEK. It is a collaboration between the Vienna University of Technology, the University of Apllied Arts Vienna and the University of Music and Performing Arts. The team is comprised of artists and researchers that cover diverse areas such as media arts, computer science, Human-Computer-Interaction, game design, musicology, ethnomusicology, technology and interface design.

PROJECT TEAM

Prof. Fares Kayali is a researcher, educator and designer living and working in Vienna, Austria. His research interests are situated in informatics, didactics, arts  and HCI with a broad spectrum covering game design and gamification, health care technology, music computing and interactive art, as well as teacher education and game-based learning. Fares has completed his habilitation in “Game Design and Education” at the University of Applied Arts Vienna and works as a senior postdoctoral researcher at the Institute of Design and Assessment of Technology at the Vienna University of Technology. There he is co-founder of the Positive Impact Games Lab (http://piglab.org) and principal investigator of the game-based learning project “Sparkling Games” and the art-based research project “Breaking the Wall - Playful interfaces for Music Audience Participation”. Fares Kayali  lectures at several Austrian universities and has presented his work in highly regarded academic publications and conferences such as CHI, DiGRA, GDC, GLS, Games for Health, Games for Change and Entertainment Computing. His works have won or been nominated for high-profile awards such as IndieCade, the Independent Games Festival, Europrix, eAward and Occursus.

Dr. Oliver Hödl works as a postdoctoral researcher and artist with a focus on Human-Computer Interaction and interface and interaction design in music. He is affiliated with the research groups Cooperative Systems at the University of Vienna and the Institute for Design and Assessment of Technology at the Vienna University of Technology. In research, he focuses on studying user experiences, art-based research approaches, and using HCI-related qualitative and quantitative research methods. His newly developed music instruments and interactive concerts have led to performances throughout Europe, USA and Australia. Furthermore, he has worked on projects around game design, urban mobility, safety and disaster solutions, and healthcare IT.

Prof Ruth Mateus-Berr Ruth Mateus-Berr is an artist, researcher and Social Designer, professor at the University of Applied Arts Vienna. Her conceptual artwork engages with contemporary global challenges such as environmental, social and political issues. Focus: linking art, design science, design thinking, health, art and design education, design research, didactics, multisensory research, artistic research, urbanism, inter-/transdisciplinarity. Recent works engaged in environment, dementia, ageism, health, urban change, democracy, racism, right-wing populism, migration, postcolonial criticism and innovative strategies on different perspective taking, interdisciplinary collaboration, participation of the audience, participatory and collaborative projects. Her personal artwork has featured in numerous exhibitions, workshops and publications in Austria and abroad.

Thomas Wagensommerer lives and works as a media artist in Vienna, Austria. He studied Digital Media Technology at the University of Applied Sciences St. Poelten, Philosophy at the University Vienna (without graduation) abd Transdiciplinary Art (TransArts) at the University of Applied Arts Vienna. He is a lecturer for Experimental Media at the University of Applied Sciences St. Poelten and a member of the research / artistic staff at the University of Applied Arts Vienna. He is an artistic assistant of the Breaking The Wall project. 

Uli Kühn lives and works in Vienna as a sculptor, media artist and musician. He also is a lecturer and research assistant at the University of Applied Arts Vienna. His work is situated on the border between music, media art, performance and experimental digital film production and has been shown at venues including diagonale graz, ars electronica, flmfest dresden, soundframe, lames, o1 kunstradio, triennale linz lentos, moozak.org, flmfest dreseden and urban art forms.

Prof. Johannes Kretz, born 1968 in Vienna, studies (composition, pedagogy) at the music academy Vienna with F.BURT and M.JARRELL and mathematics at the University Vienna • 1992/93:  studies at IRCAM, Paris with Marco Stroppa and Brian Ferneyhough • co-founder of NewTonEnsemble Vienna, of the international composers group PRISMA, of ikultur.com and of aNOther festival Vienna • teacher for music theory and composition at the conservatory of Vienna • since 1997: teacher for for computer music at the university for music and performing arts Vienna, since 2001 also music theory, since 2004 also composition, since 2009: habilitation in composition, associate professor • Since 2008: Head of ZiMT ("center for innovative music technology") of the university for music and performing arts Vienna, since 2013 dean/head of department of the Institute for composition and electro-acoustics • Founding member of NewTonEnsmble Vienna, of the European Bridges Ensemble, the international composers group PRISMA, of the performance duo TOUCHING, and of ikultur.com. Co-curator of aNOther festival Vienna together with Wei-Ya Lin und Mahdieh Bayat.

ARTISTS

Susanne Kirchmayr aka Electric Indigo, born 1965 in Vienna, Austria, works as musician, composer and DJ. She started her career 1989 in Vienna, lived in Berlin from 1993 to 1996, founded the database and network for women in electronic music and digital arts "female:pressure" in 1998 and has received several awards in Austria and Germany. She is a music composer and sound designer for the project.

Dr. Didi Bruckmayr, born 5.4.1966, lives near Linz/Austria. Performance artist, singer, actor, producer, award winning multimedia artist, scientist and dancing fool. touring the clubs and international festivals since 1985. Frequently working for contemporary opera and theatre.

Chris Bruckmayr, Mag., Sound Artist. Produces and releases dark techno music under the name raum.null on vinyl only label Belgrade dubs / Belgrade. Performances with Fuckhead & raum.null at Ars Electronica festival 2014-2016 and at the Heart of Noise festival 2016. Creative producer at Ars Electronica SPAXELS GmbH.

Master and Bachelor STUDENTS

Dipl.-Ing. Christoph Bartmann is an Austrian information scientist. He studied business informatics at the Vienna University of Technology and works as a project manager and IT-consultant. His interests include mobile applications, music, audience participation as well as data and business process management.

Bsc. Naida Comaga is a master student at TU Wien in Vienna, Austria, majoring in Media Informatics. She is currently working on her Thesis, "Audience interaction through the game based interactivity/participation" under the supervision of prof. Peter Purgathofer and Dr. Fares Kayali. Her research is based on the ubiquitous technology interfaces, human computer interaction, game design and gamification of the different interaction interfaces in a two particular areas: music and performance. 

Lukas Gartlehner is currently studying media informatics at the Vienna University of Technology. In his academic projects he focuses on the intersection of music, games and interaction design. He works as a software developer and writes his master thesis on audience participation as part of the "Breaking the Wall" project.

Alexander Filipp is a student at the Vienna University of Technology, currently studying media informatics and visual computing. At the moment, he is writing his bachelor thesis about the evaluation of music concerts with Technology Mediated Audience Participation (TMAP) with the case study “Breaking The Wall” under the supervision of professor Geraldine Fitzpatrick. On the other side, he works as a software developer and maths teacher in Austria.

ADVISORS

Prof. Geraldine Fitzpatrick, Prof. Peter Purgathofer, Thomas Gorbach  (Vienna University of Technology)
Dr. Hande Sağlam (University of Music and Performing Arts Vienna)
Dr. Simon Holland (Music Computing Lab, Open University)

CONTACT

fares <at> igw <dot> tuwien <dot> ac <dot> at