Maida Vale Live Performance Demo

This video demo combines visuals created in Unreal engine with a Dolby Atmos mix of an original composition by Jacob Cooper.

Audio Workflow

Recording

A standard multi-track recording workflow was used to record the audio. The images below show the recording setup for the drums.

Recording setup for drums.

Mixing

The audio was then mixed in Pro Tools using the Dolby Atmos workflow by Prof. Gavin Kearney. An enhanced stereo scene was presented; the core parts of the mix were presented in stereo, with the additional elements and reverbs being rendered to create a more immersive scene. Due to limitations on the track count, the drums were mixed separately and rendered to a 7.1.2 bed. The audio was rendered to two formats and exported as an Atmos ADM file:

The 7.1.4 mix was then encoded to 7th Order Ambisonics using the AmbiX 7th Order panner.

Deliverable Formats

The audio was provided in the following formats:

The BBC’s EAR Production Suite was used to convert the Atmos ADM file to an EBU ADM file. The stems were also provided separately.

Visual Workflow

The visuals were constructed using the following:

Environment Modelling

Model of Maida Vale Studio 4 [1] was created in SketchUp and exported as a .dae 3D model.

Model of Maida Vale Studio 4.

Motion-Tracking

Guitar/ Keyboard Tracking

A Vicon motion-tracking system was used to capture the performer and instrument positions for the following:

Performer playing guitar in a motion-capture suit
Finger-tracking using motion-capture markers.

The performer was captured playing along with the track. To sync this with the audio, the performer strummed along to a 1-bar count in before the track started. Full body and finger tracking was used; finger-tracking made playing near-impossible. The synth keyboard was not tracked as it remained in the same position.

Instruments were tracked using 5 motion-capture markers.

Drum Tracking - XSens

Initially, drum tracking was done using an XSens suit; sticks were tracked using markers. This didn’t facilitate the tracking of cymbals. The drummer clicked their sticks along with the 1-bar count in.

Performer playing drums whilst wearing an XSens suit.

Drum Tracking II - Vicon

The drum tracking was re-recorded to allow for the capture of moving cymbals. Instead of an acoustic kit, an electric kit was played, with markers placed on the rubber cymbals. Fingers were not fully tracked for the drummer.


Performer playing drums whilst wearing an XSens suit.

Face-Tracking

Face-tracking was recorded using the LiveLink Face for Unreal application for Apple devices (with True Depth camera). A short extract of the performer playing (< 30 seconds) was recorded and looped during the video.

3D Scanning

Instruments were scanned using an Artec 3D scanner - due to issues with reflective surfaces, the models were poor quality, and as such, were not used.

Project Assembly

The model and tracking data were combined in Unreal Engine 5; all tracking data was mapped to metahumans and they were placed within a virtual Maida Vale studio 4. Downloaded instrument assets were used for the guitars, bass and synth (keyboard).

Unreal’s ‘Sequencer’ was used to put the project together. Cameras were placed inside the virtual space to capture the video.

Download Materials

</tr>
Type Title Audio Format Filetype
Audio Only MaidaVale_Binaural Binaural
Dolby Atmos Decode
WAV
MaidaVale_7OA 7OA
ACN/SN3D
WAV
MaidaVale_7_1_4 7.1.4
Decoded from 7OA
WAV
MaidaVale_EBU_ADM EBU ADM WAV
MaidaVale_Atmos_
ADM
Atmos ADM WAV
Audio & Visuals MaidaVale_Visuals_
4k_Binaural
Binaural
Decoded from 7OA
WAV

Credits

The AudioLab would like to thank Google for their collaboration with, and funding of this project.

Composition, Recording & Production

Jacob Cooper

Mixing

Gavin Kearney

Musical Performance

Andrew Chadwick - Drums

Jacob Cooper - All Other Instruments

Motion-Capture Performance

Joe-Rees Jones - Drums

Jacob Cooper - All Other Instruments

Visuals

Joe Rees-Jones & Jacob Cooper

Asset Licences

Drum Kit: “Drum Kit” by Serg.Morgun is licensed under CC BY 4.0
Bass: Obtained from SketchFab (now removed) is licensed under CC BY 4.0
Rhythm Guitar: “Fender Stratocaster Guitar” by Ryan_Nein is licensed under CC BY 4.0
Lead Guitar: “Guitar” by Matterhorn is licensed under CC BY 4.0
Keyboard Stand: Adapted from “Keyboard X-Stand” by Michael Perez which is licensed under CC BY 4.0
Stage Lights: “Box Panel Spot Light” by Photoant77 is licensed under CC BY 4.0

References

[1] - Cairns, P, Hunt, A, Cooper, J, Johnston, D, Lee, B, Daffern, H & Kearney, G 2022, ‘Recording Music in the Metaverse’, Paper presented at Audio Engineering Society Conference: AES 2022 International Audio for Virtual and Augmented Reality Conference, Redmond, United States, 15/08/22 - 17/08/22.