Last month, Jean-Marc Jot, one of Magic Leap's resident audio experts gave a keynote at the Audio Engineering Society’s conference on audio for VR and AR . Jean-Marc, who drives our R&D around audio processing spoke about the shift in audio delivery as we enter the era of spatial computing.
Audio is used to help convincingly immerse people in entertainment content. Think of the amazing audio experiences delivered in modern theaters. Or how immersive audio helps you to tune out the real world during a VR experience. The role of the audio is to transport you to another world. "You are there."
As digital objects step out of the screen and into the real world, rather than total immersion and occlusion of the real world, audio is now fully incorporated into the user’s environment. "They are here."
Having to respect the rules and norms of the real world brings a whole new set of challenges around the audio experience. In the real world, we expect audio to behave in a certain way. For example, sounds are muffled when an object is in the way. Certain materials and shapes introduce reverb or echo. An object moving out of our line of sight gives away its position with audio.
Jean-Marc also discussed some of the technical challenges of delivering audio for completely untethered spatial computing while working on Magic Leap One. Differing from pre-recorded spatial audio, an interactive six degrees of freedom (6DoF)-based audio experience requires that audio not only responds to head rotation in a static scene, but that individual sounds dynamically “stick” to identifiable objects.
As Jean-Marc pointed out, the current audio rendering technologies for use in XR still have limitations in producing a "walkaround experience," and accurately modeling the acoustics of a real-world environment.
We’d love to hear your thoughts on audio in Magic Leap One. If you're designing a project for Magic Leap One that uses sound as a key component of the experience, let us know! Check out these learning resources for audio in spatial computing and Soundfield Audio, and feel free to ask a specific question in our forums.