A clear explanation of how depth sensing works in AR
Depth sensing is the process of measuring the distance between a device and the environment around it including objects, surfaces and people. It creates a real-time map of the surrounding world so that digital content can anchor, occlude and interact with physical space in believable ways.
Depth sensing captures how far physical content is away from the device. Different approaches and technology collect this information differently. Many devices use LiDAR sensing to send invisible light pulses into the world around it and measure how long it takes for them to return. Others use structured infrared patterns or stereo imaging to infer depth by comparing two offset viewpoints.
The output of all this information becomes a depth map, which is a grid of distance values. Frameworks like ARKit and ARCore combine this map with SLAM, World tracking and Plane detection to build a World mesh. Once the mesh is made, the system will use it to interact with digital objects through things like collision, occlusion and shadows.
Depth sensing works on all content the same including people, objects, walls and furniture. The technology makes a point cloud, identifies surfaces, and reconstructs mesh geometry in real time.
How can a system do this so fast in real time? It merges sensor data with motion tracking and refines the map every moment as a user or device moves through a space.
The result is a continuously updated understanding of the user’s environment that is accurate enough for collisions and occlusion and light enough for a device to process without causing drift or loss of frame rate.
Depth sensing allows augmented reality content to be integrated and feel seamless with the physical world. It allows digital content and models to react predictably with real surfaces and allows it to stay anchored to a space. It makes occlusion possible so virtual objects can disappear behind a column or behind a person as they walk by. It facilitates collisions so digital objects can bound, land or be aligned surfaces in your space.
Depth sensing also helps to facilitate lighting and realism in AR. When the system is aware of distances and relationships, shadows can be projected and fall with simulated physical accuracy. Mapping can also happen in real time, though some AR Scenes rely on pre-scanned environments for more accurate and precise meshes.
Quality depth sensing allows for new forms of interactions as well. Users can place models and interfaces so that they are nested among physical objects, virtual indicators can refer accurately to their physical counterparts and effects like portals can extend solid surfaces into other virtual worlds.
Designing with accurate depth can be both an opportunity and a constraint. It allows your interfaces to feel grounded, but unlike in virtual reality, you have limited space to share content.
When setting up virtual content, designers should avoid placing critical content on highly reflective or featureless areas, since depth sensors struggle to map them. Occlusion should feel intentional and designers should be aware of when their content may be unintentionally obscured by the environment. Collision should also match user expectations and elements should always appear within the bounds of the real physical space. Intentional depth and placement will improve the sense of presence of experiences and them feel authentic in AR
• A virtual sculpture that casts accurate shadows on its surroundings based on realistic scanned meshes and simulated lighting.
• An AR dragon is obscured by a pillar as it flies around the room due to accurate occlusion
• A fashion designer scans a pair of boots with their iPhone depth sensor and adds an animation Trace. This allows their audience to view a polished virtual display in AR in their own home.
• Depth sensing is a distinct technology from camera tracking. It measures physical distance not just motion.
• LiDAR is not the only method for depth sensing. Many systems use stereo or structured IR.
• A perfect mesh is not required for believable AR. Higher detail meshes may negatively affect frame rate of AR devices.
LiDAR, World mesh, Occlusion, Plane detection, 3D scanning, SLAM
Depth sensing allows digital objects to understand their environment and react to it in a realistic way. When an AR device accurately understands surfaces and physical relationships, AR becomes believable. This believability reinforces the magic of AR and enhances experiences to make them impactful and memorable.
Learn about augmtened reality or start creating your own experiences.
