Obstacle Avoidance and Interaction in Extended Reality: An Approach Based on 3D Object Detection

When immersed in Virtual Reality (VR), real objects around us could be dangerous if the users collide with them. On the other hand, such objects could be exploited to improve VR setups and to create Extended Realities where real and virtual elements are coherently mixed up, and interaction is enhanced by passive haptics. This work proposes a system that combines state-of-the-art 3D object detection, depth data from stereo cameras, and VR environment rendering. Our system can detect specific classes of objects, estimate their position and extent, and create a virtual counterpart for each one in an immersive virtual scene. We describe the pipeline of the system and show that our system allows real-time interaction with virtual objects blended with real ones.