Mixed Reality Technology – Introducing Innovative Applications

This post is also available in: עברית (Hebrew)

Mixed Reality (MR) technology has recently attracted a lot of attention with several new developments emerging on the market and some that are still at the patent stage.

Mixed Reality refers to experiences that mix augmented reality (AR) overlays with the immersion of virtual reality (VR) simulation of the environment.

Apple Inc. has been secretly weighing an expansion into digital glasses and wearable computing, according to some sources. While still in an exploration phase, the device would connect wirelessly to iPhones, show images and other information in the wearer’s field of vision, and may use augmented reality.  

According to bloomberg.com, Apple has reportedly talked about its secret glasses project with potential suppliers. Should Apple ultimately decide to proceed with the device, it would be introduced in 2018 at the earliest, another source said. Apple spokeswoman declined to comment.

The company has made several acquisitions to advance its AR capabilities. In 2013, Apple bought PrimeSense, which developed motion-sensing technology in Microsoft Corp.’s Kinect gaming system. Purchases of software startups in the field, Metaio Inc. and Flyby Media Inc., followed in 2015 and 2016.

Apple’s challenge is fitting all the technology needed into a useful pair of internet-connected glasses that are small and sleek enough for regular people to wear.

Another company, Occipital, has developed the Bridge – a new mixed reality and positional tracking headset. According to theverge.com, Bridge uses the iPhone’s camera (with the help of a lens attachment to widen the view) and mixes that video feed with the data from the Structure Sensor to create a 3D stereo view of the room you’re in. In some ways, Bridge is a lot like any other mobile VR headset: you snap in an iPhone; strap it on your head, and enjoy smartphone-powered virtual reality.

What makes Bridge unique is how it integrates Occipital’s first product: the Structure Sensor, a marker-sized strip of cameras and sensors that can map physical objects and environments with incredible levels of detail. The Structure Sensor adds inside-out positional tracking to the headset, allowing people to move around in real space without external cameras or sensors. This means that instead of just sitting in one spot and spinning around, you could actually lean your head and walk around inside a VR game or experience — something that was previously hard to come by in a mobile setting.

According to Occipital’s site, with a sensing range of over 3.5 meters and a depth map that updates at 30fps, the Structure Sensor attached to every Bridge enables robust 6-DoF inside-out positional tracking for room-scale VR experiences. Inside-out positional tracking lets you roam freely through any virtual environment, while obstacle avoidance helps you avoid objects or walls around you.

Microsoft has been another player in the field. The corporate plans to build a technology that would track users as well as the other objects in given environment and bring object tracking technology to augmented reality, as suggested by a new patent filed by the corporation. According to cnet.com, one of the challenges for more advanced augmented reality is that a system would need to track not only you as a user, but also the other objects in your environment. Microsoft’s patent document suggests that the object tracking functionality will be used in augmented reality devices to make sure you don’t forget keys, milk and other things at your home.

It will remain to see what applications these technologies would have in the security, law enforcement and military simulation fields.