A newly unearthed patent called “Gaze detection in a 3D mapping environment” shows that Apple is investigating the possibility of doing its own version of Microsoft’s Kinect, but better and with potential augmented reality applications.
The patent — reported by Digital Trends — talks about how 3D cameras would be able to track the head, arms, hands, and torso of users, but also add the ability to track the eyes in order to calculate where the user is looking at in the screen. This type of technology would allow for touchless control of any device.
While this doesn’t necessarily translate into a real product, there is a good chance that we will see a technology like this in a future product. Especially when we know that Apple is already pretty invested in 3D capture — the frontal TrueDepth cameras in the iPhone X series and new iPad Pro can track your face with high precision in three dimensions.
The most logical application would be in a future Apple TV update. Just like Kinect allowed you to control your Xbox with motion gestures, Apple may implement this tech to allow you to control your channels and apps without having to touch a remote control. In fact, the patent talks about how this technology could replace these devices.
The other application would be to build this into Apple’s desktops and laptops, as well as the iPhone and iPads. In the case of the latter, the application would be limited to the eyes tracking, as the hands would be busy touching the screen.
For the computers, however, it’s easy to imagine a Minority Report-style interface. It seems like a remote possibility, however. One that Leap Motion tried and failed with its desktop hand tracking product (more on Leap Motion below).
Tracking hands and eyes for AR experiences
But the most appealing application for this type of technology may be in a yet-to-be released product: augmented reality glasses.
We know that Apple is clearly working on AR glasses, not only because Tim Cook has declared AR an strategic future technology for the company but because the string of third-party AR companies acquisitions in the last few years is too big to ignore.
But amongst these buys there was a failed bid: Leap Motion. Now we know that Apple has tried to unsuccessfully acquire the hand and finger tracking technology company not only once, but twice. Here’s a demonstration on how Leap Motion tech tracks hands and fingers with precision:
This type of technology is crucial for the success of a seamless AR glasses product. You can’t depend on external accessories, like the Oculus Rift does with its controls in the VR space.
Manipulating AR objects in real space has to be transparent to the user, and that will depend on precise hands, eyes, and body tracking.
Leap Motion’s tech does a lot of that already. It is similar to the one described in this patent, except Apple expands it to gaze tracking.
And while Apple’s patent describes external cameras, flipping them around and building them into the glasses themselves will work just the same. Tracking your eyes with cameras mounted on the frame wouldn’t be hard either. You just need the adequate miniaturized components and optics — something we know Apple is already working on through its acquired companies.
Of course, perhaps all of this is just a pie in the sky dream, and this patent will never reach the market. But everything seems to be aligning to make the Next Big Thing a reality.
