Zach Kinstner is the founder of EXA, an immersive musical studio where you can compose, record, and perform music using instruments designed and built by you. We caught up with Zach to find out some of the challenges of adapting his popular VR experience for spatial computing.
Zach Kinstner, the founder of EXA and recipient of our Independent Creator Program grant, has now launched “EXA: The Infinite Instrument” on Magic Leap World.
EXA is an immersive musical studio where you can compose, record, and perform music using instruments designed and built by you. So you can sit at a piano and have the EXA “virtual band” accompany you. Or your band could have an “EXA player” play alongside your guitarist, bassist, and drummer.
We caught up with Zach to find out some of the challenges of adapting his popular VR experience for spatial computing, and the lessons he learned along the way.
Zach, what got you into spatial computing?
I started to fall in love with spatial computing around 2016, and I haven’t looked back. The creative possibilities in this space are endless, and I enjoy exploring it. I’ve focused on user interfaces and interactions in immersive applications — seeking out new solutions and bringing everything together to improve the overall experience.
Even though EXA began as a virtual reality application, my prior experience working with augmented reality was an important influence on the app’s overall design. I didn’t have specific plans to create a spatial computing version of EXA when I created the VR application, but I did make decisions throughout the development process that left that possibility open.
While I was building EXA, Magic Leap One launched and I was excited about its potential. When the Independent Creator Program was announced, I jumped at the opportunity, and I was fortunate enough to be selected for it.
What were some of the challenges you faced?
EXA is a complex application with some unique requirements, and this made the technical aspect of the conversion quite challenging. EXA uses a custom, native (non-Unity) audio system to achieve very low-latency audio. This required a deep dive into Magic Leap One’s audio API and a special compilation process to target the Lumin OS. Also, although EXA performs quite well in VR on a desktop computer, it needed significant performance optimization for Magic Leap One. I spent nearly two months doing nothing but optimization work; in particular, learning Unity’s brand new DOTS architecture, and refactoring everything I could in order to take advantage of its performance benefits.
I’d guess that most VR apps would need at least a little optimization for Magic Leap One, but many would avoid some of EXA’s biggest challenges. EXA’s size, scope, and technical requirements just happened to push the conversion process into very difficult territory.
What can you do in spatial computing that you can't in VR?
I’m most excited about using EXA in combination with other real-world instruments. You could, for example, sit at a piano and have the EXA “virtual band” accompanying you. Or your band could have an “EXA player” playing alongside your guitarist, bassist, and drummer.
Magic Leap One makes these use cases possible because it maps digital content to specific positions in the physical world, and allows you to see both digital and physical content at the same time. You can also take Magic Leap One almost anywhere, with very little setup required.
What was it like developing for Magic Leap One as a solo dev?
The biggest factor, of course, is that all responsibility falls to you. Development, design, planning, testing, debugging, fixing, communication, decision-making, brainstorming, marketing, and more — if you don’t do it yourself, it doesn’t get done. That’s a heavy load to carry, but it can make the process more efficient. For example, I can usually visualize how I want my project to look, behave, and feel. Working solo, I can move directly into implementation, rather than trying to document and communicate it all to a team.
Another important factor involves handling the new and unknown aspects of a project. In a team setting, you might bring in a specialist, or appoint a team member to tackle a new task. But when you’re working solo, you have to take time away from known tasks, and devote that time to research and learning instead. For example, I knew nothing about Unity’s new DOTS architecture before this project. Once I’d determined that it was my best option for improving performance, I set aside time to learn it by digging through documentation and figuring out how to refactor my existing EXA code. I also gained a useful new skill during the process, which I’ll be able to apply to future projects.
How do you see the act of creating music and performance evolving with spatial computing?
Some of the benefits that spatial computing already has to offer music and performance is that it immerses you in a flexible, customizable, music workspace. Players have the freedom to build and shape instruments to suit their own needs.
The ability to watch and/or perform with others in real time, via multiplayer sessions, is quite powerful. You can share a musical workspace with friends from across the globe, see each others’ movements, share instruments and recordings, talk to each other, and create together.
Hardware improvements can also improve music experiences with highly accurate hand and finger tracking, haptic capabilities to simulate recoil and resistance from instruments, more methods of input to control sounds, and so on.
What gets you excited about the future?
After working with VR tethered to my computer for so long, one of my favorite things about Magic Leap One is that I’m free to simply walk into a different room. Also, the entire kit fits into a Carry Case, making it possible for you to bring EXA and your other Magic Leap One apps almost anywhere, with minimal setup once you’re ready to play.
What advice would you give to other developers creating spatial computing experiences?
One of the most useful things you can do is to constantly seek out feedback, issues, and ideas from real users. Not only are there no universal “best practices” for spatial computing apps, but you’ll quickly find that users will move, look, and act in ways you never expected.
Want to start developing for spatial computing? Join our global community at our Creator Portal.