Spatial Mapping for Magic Leap 2
Last Revised: September 29, 2022
In this Learn More, we cover:
- What is Spatial Mapping?
- What About People in My Surroundings?
- Elements of a Space
- Device Localization
- Types of Spaces
- App Permissions
For information regarding Magic Leap 1 devices or Magic Leap’s first generation ecosystem, please refer to the “Learn Mores” for Magic Leap 1, available here.
What is Spatial Mapping?
Spatial maps are a critical part of the Magic Leap ecosystem as they help Magic Leap enable interactions between digital content and your actual physical surroundings. Magic Leap 2 device sensors scan your environment, process that information, and use it to create three dimensional representations of your area (which we refer to as Spaces). The end result is a mapped environment that enables apps to render digital media in a user’s field of view as if the content is really in front of the user.
When a user utilizes the Spaces application to scan an environment with their Magic Leap 2 device, computer vision algorithms use images collected from the world sensing cameras on the Magic Leap 2 device to extract features from those images and create a representation of surroundings in that environment. The final set of world-understanding data is a spatial map, or Space, at its most basic level. The processing to extract this data from world sensing camera images is done locally on your Magic Leap 2 device.
What About People in My Surroundings?
The data captured and included in a Space only reflects information from the parts of your environment that are stationary. If a person walks through the room while a Magic Leap 2 device is scanning the space, the person will be excluded from the Space. If a person is sitting in the room while the Magic Leap 2 device is scanning the space, that individual may be included in the Space created, but the Space is not designed to reflect a level of detail sufficient to identify that individual. Scanning does not distinguish between organic and inorganic objects, just between stationary and moving objects.
Elements of a Space
The individual elements of a Space are feature points, dense mesh, spatial anchors, planes and positional data. Each of these combine to provide a way for the Magic Leap 2 device to understand where it is in an area, the geometry of the area, and where content is placed.
Feature points are unique architectural and texture-related points in the world that help the Magic Leap 2 device distinguish an area. Examples include edges of windows or picture frames, textured walls, tabletops and doorway molding. Feature points are an underlying structure of the Magic Leap 2 device’s world understanding, but are not exposed to users on the device’s display. The more feature points in the view of the device, the easier it is to localize. Some materials, such as glass, black or glossy surfaces, do not render feature points.
Dense mesh is the 3D triangulated geometry that represents the Magic Leap 2 device’s understanding of the real-world geometry of an area such as a room or office. During a scan session, the Magic Leap 2 device will create an overlaid pattern to represent the 3D geometry. The longer a user looks at a specific area, the more detailed the dense mesh. Dense mesh may be used to place content on top of a table or against a wall, obstruct content that is behind walls or desks, or provide collision for objects. The dense mesh scans stationary objects, while moving objects, such as people walking through an area, are not scanned. There may be “ghost” geometry present if an object moves slowly across an area being scanned.
From dense mesh data, planes (or large flat surfaces) can be derived. This helps identify walls, floors, and optimal surfaces for placing content. Dense mesh data and planes additionally enable occlusion and collision consistent with your environment.
Spatial anchors are used for the persistent placement of content. Each Space includes a single spatial anchor to provide the Space’s zero-point. Additional anchors may be added when a user places content when in an application or a developer may want to include more anchors throughout a Space to provide consistent content across multiple sessions.
Capturing positional data during the scanning process is essential for consistent device localization. During a scan, the user is directed to turn in a 360° circle every 2m/sq. This sequential set of movements captures special key rig frames throughout the Space. The more key rig frames captured, the more likely a Magic Leap 2 device can consistently localize into the Space from any place within the real-world area. During the scanning process, a user will be taken through these steps to ensure quality device localization.
Device localization is when the Magic Leap 2 device can recognize enough of a real-world area to understand where the Magic Leap 2 device and user is within the Space. When device localization occurs, content can be placed, and stay in place, relative to where it is in the world. Solid localization is based upon a scan that offers quality positional data and feature points.
Ad-hoc localization, however, does not require any pre-mapping or scanning; it runs in the background where users may not be aware of the process. Ad-hoc localization allows users to place world-relative content, but that content will not persist across sessions.
Types of Spaces
There are two types of Spaces: local and shared. Local Spaces exist only on a single Magic Leap 2 device and offer persistent content placement. Local Spaces do not offer colocation of content between multiple devices. Shared Spaces utilize AR Cloud and offer persistent content across sessions, while also offering large-scale Spaces and colocation across multiple devices. Both Local and Shared Spaces are created using the Spaces platform application.
User permissions are required for spatial maps to be accessed by applications on your Magic Leap 2 device, including the ability for apps to use dense mesh data, and plane detection. Apps need this information to display within and interact with your environment. If a user grants an app permission to access their Space then please note that such apps will be able to access the spatial data of their surroundings while such apps are running on your Magic Leap 2 device, even while in the background. Learn more about the choices you may have with respect to the data that apps on your Magic Leap 2 device may have access to here.
Spatial Mapping Storage Options
Magic Leap currently offers two storage options depending on the type of Space: on-device storage for Local Spaces and AR Cloud for Shared Spaces. For Local Spaces, you will be able to scan and store on your Magic Leap 2 device up to 5 Spaces (up to 100m2 in a single Space).
For Shared Spaces, Magic Leap offers enterprises the option to store on-cloud using AR Cloud, an enterprise-managed cloud repository. Learn more about AR Cloud and how data may be collected, processed, and/or shared by your managing enterprise here.
As described above, Spaces utilizing ad-hoc localization do not require on-device storage; they are continuously created and overwritten as needed on your Magic Leap 2 device.