Ahead of the SXSW Interactive Innovation Awards Ceremony, we caught up with the team that worked on Tónandi. The Magic Leap Studios team is one of 5 Finalists in the Visual Media Experience category. Winners will be announced at the Awards Ceremony tonight–Monday, March 11
The Magic Leap Studios team took some time to share how they brought the celestial realms of Sigur Rós’ Tónandi to life. We spoke to Mike Tucker and Stephen Mangiat, creative and technical leads of Tónandi from our Magic Leap Studios team, about the process of making musical lifeforms, designing for adaptive worlds and crossing the spatial (and supernatural) divide.
Congratulations on your D.I.C.E. Award for immersive achievement and your nomination for the SXSW Interactive Innovation Awards. The design process behind Tónandi must be an interesting one—how did it all start?
Mike Tucker: Ironically, the inception of the project dates back long before any of our team members were even employees at the company. Rony, being a big fan of Sigur Rós, invited them to the office in the early days. Together with the band’s Creative Director, Sarah Hopper, they dreamed up an idea of “sound spirits,” literally translating to “Tónandi” in the Icelandic language.
It wasn’t until years later that the hardware was ready. We spent some time interpreting the briefing: How do these sound spirits behave? Do they interact with your environment, each other, the user? We arrived at the analogy of an ecosystem of life forms; ranging from plant-like to animal-like, more literal creatures to more abstract.
“It was a fluid collaboration: a constant cycle of music informing visuals informing interaction.”
Mike Tucker: Later on, when our “Product Equivalent” of Magic Leap One (ML1) began to mature, we put a proper team together. The Mill’s Emerging Tech team worked with us on developing some of the creatures, elements, and engineering efforts. Sarah Hopper represented visuals from the band and Paul Corley, the band’s Music Director, curated and developed new sounds for the experience. From then on it was a fluid collaboration: a constant cycle of music informing visuals informing interaction, on and on.
Stephen Mangiat: When we started developing, Tónandi was an experience that did not exist for a platform that did not exist. This was daunting, exciting, and at times scary, but ultimately it gave everyone a truly unique opportunity. One thing that was always there was a belief that Sigur Rós and their music and aesthetic were right for mixed reality and what we were hoping to achieve.
Following on from that initial idea, what was it like working with Sigur Rós on the evolution of the Tónandi experience?
Mike Tucker: It’s honestly quite amazing how well the original concept held up after all these years. Often a project may take some twists and turns before a golden idea is found, but the elegant simplicity of a “sound spirit” was able to carry us into many interesting areas. On the music side, there was a tremendous sense of trust. Their work was chopped up, re-pitched, looped, and combined in so many different ways—sometimes it was hardly recognizable. I can’t think of many musicians who would be so open-minded with this process.
Stephen Mangiat: Music creation was core to the concept - the idea that you could have access to musical tools the band would use in the studio. However, we knew it shouldn’t be a game, like hitting a virtual drum pad in time with a beat. This meant diving into their unique composing process, and analyzing the components of songs and the tools used to make them - any fan’s dream!
Many Sigur Rós sounds are created through a process of experimentation and discovery. For example, one tool that the band had been exploring is a sample arpeggiator, which allows you to create rhythmic patterns using short recorded samples, often of Jónsi’s voice. Traditionally, they would modify these patterns using knobs and sliders, but we abstracted these controls into stones that you can move in 3D space using magnetic repulsion to your hand.
The system itself is not presented to the user, but through a visual abstraction and natural interaction you could work out that the placement of the stones will affect the samples and rhythms. You can then move them around until you find something particular pleasing—and in essence, you are composing using a method similar to the band.
It was a lot of fun and a huge benefit that the band was so open to modifying their samples directly (rather than synthesizing sounds in the engine), as it was critical to making the world sound authentic.
How did you approach giving physical shape and behavior to these "sound spirits"?
Mike Tucker: This might have been the most challenging aspect of the project. We wanted the visuals, music, and interaction to have equal influence on each other. For instance, the Flock began as a visual prototype, simply a school of fish. Next, we said, “How does this behave?” We arrived at having both a calm and agitated state—triggered when the user moves their hands through it. Finally, we created a granular synthesis system which controls dozens of short sound samples called grains, each spatialized within the mass of fish.
In another case, we did the exact opposite. Steve wanted to capture Jónsi’s iconic “bowed guitar” sound—extracting loops from Jónsi’s performance at a variety of pitches. Those were then mapped to a grid of blocks which could be raised and lowered by your hands. The final piece would be the visuals. How do you communicate to the user this underlying system of blocks, but convey it in a beautiful way matching the aesthetic of the experience? From there, we iterated on some visual treatments before arriving at the crystal-like shards.
Stephen Mangiat: Many of the creatures utilize particle effects, which are often minimal and use additive light blending. These look great on the display - like creatures made of light - but they also allow for more responsive and dynamic interactions. For example, particles can disperse in interesting patterns as your hand passes through them, like smoke or fog.
Particles also allowed us to visualize sound directly without any abstraction, as in the case of the “light seed,” the ball of electricity that you touch to start the experience. The light seed was one of the first prototypes developed for the project, and its power is in the simplicity - we draw the sound wave in 3D space, it warps to your finger as you approach it, and this modifies the sound wave and thus the sound you hear. That’s it! You are touching the sound wave and the visceral reactions we got early on encouraged us to keep exploring that paradigm as the experience developed.
How did you structure and sequence the “worlds” to give each spirit their unique personality?
Mike Tucker: Each creature at its essence is a combination of visual, sound, and interactivity—meaning they could be given a color scheme or musical key. This allowed us to have full control over how we wanted to mix them together, which we ended up calling “Recipes.” Thinking of each creature as an ingredient, we began to experiment with what combinations would yield the most interesting musical and visual compositions. Because the two were inherently dependent on the other, it was never easy to say, “Oh we’ll just remove or add this” because of the impact it would have on everything else in that Recipe. Much in the same way you can’t throw ice cream together with pizza and expect it to taste good.
Each Recipe consists of events on a sequential path, with some potential to go down alternate paths through interactions by the user. Each element along that path is an opportunity for some randomness and it was practically a daily debate how much should occur. We hope future releases of Tónandi will introduce more variety to keep people coming back to the experience.
One of the most exciting aspects of Tónandi is its semi-random, adaptive nature. What have you learned about creating procedurally generated spatial experiences?
Mike Tucker: Capturing an understanding of the environment proved to be a lot trickier than anticipated. Simple questions like—“What is the center of the room?”—are surprisingly difficult if the user can literally be anywhere. With Tónandi we chose to embrace this variety by designing for randomness.
Under the hood, we have a set of parameters and constraints for each creature. For example; some are defined to only be created on flat surfaces, some may only exist on ceilings, and others may require a certain amount of empty area. This allows us to fine-tune how the experience will feel for a variety of environments. Many of these challenges will be common to developers and we’ll likely be seeing a number of solutions become available as the medium matures.
“It completely changed our design process—opening up possibilities in simulating physical characteristics and interactive animations on a huge scale.”
Stephen Mangiat: Procedural generation is really hard, particularly for visuals that must respond to any environment. To aid the variety we used spatial sound techniques like random spawning, with random samples, timing, and positioning. We could also move the sounds through space, like rocks falling from the sky. Another key component brought by Paul and the band was the use of natural ambient recordings from Iceland, recorded in caves or beneath glaciers. These added an aural variety that is difficult to achieve with instruments, and were critical to making it feel like you are immersed in a “space.” Also, who knew that glaciers sound so cool?
Tónandi is visually stunning, but seems like it could be quite demanding on the GPU. Are there any tips you'd like to share for creating graphically ambitious experiences on the Magic Leap One?
Mike Tucker: When we began this project, one major goal was to push the tech in every way possible—this meant the interactions, visuals, audio and getting as much out of the CPU and GPU as possible. Initially, the project was very much CPU heavy. Having dozens of spatialized sounds playing simultaneously was a big contributor. Meanwhile, the GPU was hardly being leveraged.
Our Lead Engineer, John Cannon, joined the team around the time we began working with the “Product Equivalent” hardware. He introduced us to Compute Shaders, a fantastic method of leveraging the power in our GPU. It completely changed our design process—opening up possibilities in simulating physical characteristics and interactive animations on a huge scale. Elements like the vegetation, vortices of light, and flocks of fish would not have been possible without this new way of creating content.
Check out the related L.E.A.P. talk below:
Lastly, Magic Leap One’s meshing, spatial audio and powerful GPU are changing the way people can experience music. Does this mark a new, immersive era for creators?
Mike Tucker: Since the introduction of vinyl records, we have been tied down to the arbitrary constraint of replayable sound files. This trend continued through CDs and now streaming services. Nowadays we all carry computers in our pockets. Tónandi, along with projects like PolyFauna, Bjork’s Biophilia and Brian Eno’s Bloom can be seen as steps toward embracing non-linear music on emerging platforms. Musicians often try things differently while performing live: making each performance unique in some way. Why can’t that be experienced elsewhere?
“With Magic Leap, we have an opportunity to redefine how music can be experienced.”
Mike Tucker: With Magic Leap, we have an opportunity to redefine how music can be experienced. Features like Spatial Audio bring us beyond typical stereo mixing. Meshing the environment allows us to create compositions that adapt to our surroundings. Hand Tracking allows us free users from the cumbersome process of control schemes. All these features are at the disposal of new creators, and we hope Tónandi serves as proof and inspiration that there is much yet to be explored.
Stephen Mangiat: Tónandi is visual music, but also a small step toward music that is aware of you - aware of your hands and body and how they move through space, aware of your eyes and where they are looking, the time of day, what environment you are in, etc. This awareness could be a powerful tool for musical composition, and perhaps even fluidly adapt to your emotional state. What happens when your music knows you’re at the gym and getting tired, or it knows that you’ve had a long day at work? Making listeners feel more like collaborators with composers has been a big motivator for us, and it will be exciting to see where creators go in the future.
Explore The Worlds of Tónandi
We’re over the (virtual) moon to be nominated for another technical achievement award in partnership with Sigur Rós. For the next gen of creators, musicians and makers, the gates to the world of immersive and spatial design are very much open.
We’re all about changing reality with immersive content for Magic Leap One. If you want to go deeper into your own design, there are heaps of articles in our Creator Portal to check out. And if you want to experience the spirit of Sigur Rós for yourself, Tónandi will always be waiting ... (and ready right now, but that sounds less mystical).