Scalable Audio Frameworks For Immersive Realtime Experiences

Project Profile

Project Description

The AudioLab at the University of York, in collaboration with BBC R&D and the BBC Singers, presents the next generation immersive and interactive networked performance experience for the Metaverse. This transformative project has delivered a novel framework where performers at remote locations can come together in a shared virtual performance space. This is completely scalable, and technologically democratic for use with a mobile device and headphones. BBC Maida Vale studio 2, the home of the BBC Singers, and a venue of significant musical heritage has served as the acoustic for the experience. A pilot was showcased at the Audio Engineering Society Spatial Audio Conference in 2023, followed by a public virtual mass choir workshop, culminating in a newly commissioned piece to be performed at the BBC Singers’ centenary concert in London’s Barbican.

Project Details

SAFFIRE Concert: Distributed Ensemble at AES Huddersfield 2023

A live, networked choir concert was performed between two sites: Phipps Hall, University of Huddersfield and AudioLab, University of York. Both sites had four performers (soprano, alto, tenor and bass) and a live audience.

Phipps Hall at Huddersfield contained four loudspeakers, each representing a remote performer in York. Auralisation of both local and remote performers was rendered using SRIRs recorded in the BBC’s Maida Vale studio 2 and played back via the Huddersfield Immersive Sound System (HISS) using an Ambisonic workflow. This was delivered as part of the 2023 AES International Conference on Spatial and Immersive Audio to a live, seated audience.

The AudioLab at the University of York housed four performers in a small studio. Headphone monitoring of both sites was set up in an adjacent room with the same auralisation rendered over headphones.

The two sites were connected over the internet using the existing JANET network infrastructure. Audio was streamed between sites using JackTrip and a full system throughput latency of approximately 16 ms was achieved.

Following the concert, the audio materials were mixed using an Ambisonic workflow in Reaper by Tomasz Rudzki and Jacob Cooper and uploaded to Zenodo in 7th Order Ambisonic and binaural formats.

Sing Live with the BBC Singers (May 2024)

In April 2024, members of the public were invited by the BBC and AudioLab to take part in a live virtual choir event ‘Sing Live with the BBC Singers’ that took place on 21st May 2024. This event allowed performers to join in with a BBC Singers rehearsal via a bespoke application developed at the University of York.

The application allowed users to select their part within the choir, and delivered a binaural audio and video feed from that perspective to their device. It facilitated two way interaction between the conductor and participants, by allowing participants to react with a series of emojis that were fed back to the conductor. It also allowed users to record their contribution to the choir live during the event; the instruction video below shows how the application worked.

The application was designed to be simple and user-friendly, to ensure that experience with technology did not limit the number of participants. Live tech support was successfully delivered in advance of and during the masterclass.

The technical setup facilitated four audiovisual streams (soprano, alto, tenor and bass). This involved a picture-in-picture setup with a close-up view of the conductor and a wider view of the choir. The audio was a combination of a KU100 dummy head at the selected location and a DPA on the conductor. This DPA was auralised using a Spatial Room Impulse Response (SRIR) of Maida Vale studio 2 that was decoded to binaural. Video streaming was delivered via OBS to Amazon Web Services (AWS) and played back via the application.

Metadata embedded within the application’s streaming functionality allowed for post-production checks for audio dropouts to be conducted. Manual clapping checks also were implemented within the masterclass to allow for sync in post-production.

Over 200 participants took part in this event and around 75% of them chose to upload their recordings via the application!

Feedback from Participants

“Absolutely fantastic – as a long time member of stay at home choir and having uploaded tens of recordings, this was a brilliant new way to do it and to feel really part of a choir at the learning and recording stages.”

“It was fun to be able to do a recording “live” like this app, rather than the somewhat laborious process of setting up to do my own recordings at home.“

“I loved the experience ! I loved the conductors warm up exercises. I felt a part of the group.“

BBC Singers’ Centenary Concert (October 2024)

The recordings from the event are currently being post-processed and prepared for the BBC Singers’ Centenary Concert on 2nd October 2024. Tickets are available for this event online.