Big changes are coming to your phone's smartphone camera next year, with Qualcomm previewing an update to its image signal processor (ISP) that will better support features like face recognition and mixed reality.
Qualcomm's Spectra ISP is a part of the Snapdragon system on chip that's a popular mobile processor platform for many Android phones. While the next major Snapdragon update won't arrive until next year, the changes planned for the Spectra ISP have major implications not just for the cameras on 2018 Android phones but for virtual- and augmented reality headsets as well.
That's because the next version of the Spectra ISP introduces a new architecture to support advances in image quality, image recognition and power efficiency. Specifically, Qualcomm is promising that its new camera module will feature improved biometric sensing for detecting people's faces and support for depth sensing that can power mixed reality features for smartphones and headsets.
New Camera Modules
To that end, Qualcomm is adding three modules to its camera setup — one for iris authentication and a pair that add passive and active depth sensing.
The iris authentication module provides always-on security that can support phone unlocking features. Qualcomm says the sensor will be able to work in low light and through most sunglasses. If iris authentication works as promised, you won't be able to fool it with pictures of eyes or even 3D molds of faces.
Iris scanning is becoming an important feature on mobile devices, and not just as a way to augment other security features like fingerprint unlocking. As smartphone screens have gotten more expansive — think Samsung's Galaxy S8 or LG's G6 — smartphone makers have had to remove physical home buttons from the front of their phones that could double as fingerprint readers. Improved iris scanning gives those devices another way to securely unlock a phone that saves users from having to fumble to find an awkwardly placed fingerprint sensor on the rear of the device.
All About Depth Sensing
It's the pair of depth-sensing modules that have even greater implications for next year's phones and headsets, though. Depth sensing drives a number of different features like improved image quality on bokeh effects for photos, more powerful mixed reality apps and better facial scanning.
The passive depth sensor Qualcomm's adding will use differently placed cameras to capture two different images of the same object. By calculating the disparity between those images — the larger the disparity, the closer the object — the depth sensor can determine how far away that object is.
Qualcomm plans to include the passive depth sensor on what it calls "Snapdragon value tier devices." That means depth-sensing will be available to more modestly priced phones in 2018, just as dual rear cameras found their way into mid-tier and budget phones this year.
But passive depth sensing has its limits. It doesn't work so well in low light which can result in a noisier image map. It also has trouble in rooms where the walls and floors don't have a pattern.
That's where the active depth-sensing module comes in. That module uses infrared illuminator and camera to collect information about an object, collecting more than 10,000 points of depth to create a high-resolution map that can be manipulated in real-time. Qualcomm says its module can detect changes in depth as small as 0.1 mm apart.
To show how that works in practice, Phillip Jacobowitz, Qualcomm's lead on camera product marketing, showed me footage captured by the new Spectra ISP of someone playing the piano. The perspective of the image shifted easily from an overhead view to a sideways look at hands hitting the keys; eventually the camera could even show the hands separating from the rest of the body. That illustrates how this active depth sensing technology could be used to help with hand tracking, Jacobwitz said, eliminating the need for physical controllers in AR and VR. It's also easy to see how the role that kind of imaging prowess could play in instruction and diagnosis.
Other ISP Improvements
Those modules may highlight some of the changes Qualcomm has planned for its ISP, but they're not all the enhancements on tap for the next version of Spectra. If you shoot a lot of video on your phone, you'll appreciate Spectra's support for motion compensated temporal filtering (MCTF), which will allow for sharper images of moving objects, even when the camera itself is moving. Jacobwitz showed another video, this one of a busy city street festooned with lights shot from a drone. The footage captured with MCTF support had less noise, so it was sharper and more colorful, with even background details visible in the low light.
Other features planned for the second-generation Spectra ISP include accelerated electronic image stabilization, machine learning-powered face detection and simultaneous localization and mapping that will aid AR and VR programs.
We'll have to wait until next year to learn about all the new features in the Spectra ISP and the next-generation Snapdragon processor it will ship with. But from what Qualcomm has shown off so far, expect mobile photography to make a big jump in 2018.
