We’re closer than ever to seeing how Microsoft's HoloLens could change the way we work, study and play. First announced in January 2015, the augmented-reality headset is now in the hands of developers, who are working to build programs for Microsoft’sholographic platform that could make AR a mainstream phenomenon.
We’ve had the chance to spend some up-close time with HoloLens during the last two Build conferences Microsoft has held for developers. That’s given us a chance to build interactive holograms, walk on the surface Mars and see what Skype calls might look like through an AR headset. We’ve also gotten a better idea about the features of HoloLens and the Windows Holographic platform. Here are 11 important things we know about HoloLens and five things we’re still waiting to find out.
WHAT WE KNOW
1. HoloLens is a new breed of holographic computer
Microsoft's HoloLens is a very sophisticated piece of hardware, featuring multiple sensors, high-definition see-through lenses and an advanced optical projection system to generate holographic objects. An inertial measurement unit (IMU) — which includes an accelerometer, gyroscope and magnetometer — is paired with cameras to track your head and its movement.
A CPU powers the HoloLens. Windows Central ran the AID64 Mobile hardware information utility on the HoloLens and found that the headset uses a 1.04GHz Intel Atom x5-Z8100 processor. Microsoft also created a dedicated holographic processing unit (HPU) that knows where you're looking, interprets your gestures and spatially maps your environment.
That Windows Central report also found the HoloLens uses 2GB of RAM and offers 64GB of storage. There’s a 2.4-megapixel camera on the headset that can also shoot video.
MORE: HoloLens Puts the Future of Computing on Your Face
2. There are five ways to interact with holographic objects
During our hands-on time with the HoloLens, we’ve learned about the ways users interact with the headset:
a. Gaze – The software knows where you're looking and uses that as navigation. During one of our demos, when I made a sample app, a red circle followed my eye movements, similar to how a pointer follows a mouse. The red circle isn't standard, though. Developers will be able to use any size and shape of pointer they wish.
b. Gestures – Developers will be able to add their own custom gestures to apps, but Microsoft has provided some standard ones so that users don't have to learn a completely new set for every app. The clicking gesture is very simple and intuitive. You simply stick up your index finger and then lower it until it hits your outstretched thumb.
I was blown away by the HoloLens' ability to detect my gestures even when my hands were not in the field of view. For example, when I put my hands at my side and "clicked," the clicks registered perfectly for both hands. Only when I put my hands behind my back did the system fail to detect them. When I clicked on each of the spheres in the demo environment, the object fell down from its floating position and rolled onto the floor.
c. Voice recognition – It almost goes without saying that a wearable device would do voice recognition. I coded my demo app to cause each sphere to fall when I had the pointer over it and said, "Drop sphere." Saying "reset world" brought the app back to its default state, with both spheres in position.
d. Spatial audio – Though it doesn't come with earbuds, the HoloLens has built-in speakers and even volume controls. When sound is attached to an object, it will grow louder as you move closer to that hologram and softer as you walk away from it. I assigned some simple background music to my app and found it got louder as I got closer.
e. Spatial mapping – The software knows about every real-world object in the room and can let you place virtual objects on top of them. I had fun moving my canvas around the session room by clicking on it, dragging it with me by moving my eyes and then clicking again to set it in place.
I placed our canvas in midair or on top of a table and couch. Interestingly, when I put the canvas on the floor with half of it underneath a sofa, the entire block rose to the level of the furniture rather than half disappearing under it. Microsoft told me that, in other applications, developers can make a virtual object appear to be on multiple levels or angled surfaces, like rocks.
And just as volume changes relative to your distance from an object, so can the size of an object. During a demo at the 2016 Build conference, attendees could fling holographic balls at virtual targets: as the ball hurdled across the room, it became smaller and smaller.
While I was trying out the special mapping feature, I set the room's grid to be visible. As I looked around at furniture walls and even people, all of them were covered in lines, because this is how HoloLens sees the world and knows where to put objects.
Though Microsoft isn't highlighting the feature, the company also said that HoloLens can pair with Bluetooth input devices like keyboards and mice.
3. Holographic apps are Windows 10 apps
Since 2015, Microsoft has been stressed that Windows Holographic apps are standard Windows apps that use the Holographic API. In fact, a developer could build a universal app that works on desktops and tablets, but then adapts to AR when you put on the HoloLens glasses.
4. Holographic apps are written in Unity and C#
To create Holographic apps, programmers will need two tools: Unity, one of the leading 3D graphics and game development environments, and Microsoft Visual Studio. During one of our demo sessions with HoloLens, I used Unity to create a project and drag and drop 3D images onto our canvas, which had a flat, rectangular surface with some shapes on top of it and spheres floating about it. I controlled the look and feel of the cursor and which code files would be used to provide interactivity (sound, physics, etc.).
However, the code files themselves were all written in C#, which most people edit in Visual Studio. Also, to compile and test our app, we had to export it from Unity to Visual Studio.
5. HoloLens has a narrow field of view
Although I was very impressed with HoloLens overall, I wish it had wider viewing angles. You can see the entire world in front of you through the visor, but the actual augmented-reality view doesn't take up your entire field of vision. When I looked at holographic objects, the edges were often cut off if I moved my eyes too far up or down or got too close to an object. The holographic world still takes up most of the space in front of your field of view, but it often looks like a letterboxed movie, where the AR content takes up the middle of your field of view but the top and bottom of the visor can't show the objects. Contrast that with the Meta 2 AR headset, which boasts a 90-degree field of view and, as a result, doesn’t suffer as much from cut-off objects.
6. The software is in great shape; the hardware takes getting used to
As earlier as the 2015 Build conference , the software part of the HoloLens project looked like it was good to go. The hardware needed some work — our demo at the 2015 conference featured a lot of defective headsets that needed to be swapped out — but Microsoft has made the device a bit more durable. When we tried on the headset at the 2016 Build conference, the outer plastic ring felt a little bit more flexible than earlier versions.
The HoloLens remains a little bulky, and you never really forget that you’re wearing a headset. You adjust the fit of the HoloLens by twisting a dial on the back to tighten an inner band that fastens the headset in place. When I wore it, the device fit snugly but felt really heavy.
Good news if you wear eyeglasses, though: Microsoft says the adjustability of HoloLens allows it to fit over the majority of eyeglass frames. It certainly fit over my glasses with no problem, but it was a bit tight, and the bridge of my glasses got pushed into my face.
7. Developers now have the HoloLens on hand
The HoloLens developer kit started shipping at the end of March to developers who paid a pretty penny — $3,000 — to get their hands on the headset. You have to be a Windows Insider in either the U.S. or Canada to order a HoloLens, and developers are limited to ordering two. By making the headset available (along with source code for some already created apps), Microsoft is hoping to inspire developers to create apps of their own that will be ready for when HoloLens becomes more widely available.
8. Multiplayer gaming will be supported
Fans of Halo and other popular Xbox multiplayer games might not be on their own come launch day. The Xbox Live API, which allows programs and games to hook into the console's online platform, is available to developers making content for the augmented reality headset. That said, the focus won’t be on games the way it is with virtual reality headsets like the Oculus Rift and HTC Vive. "The primary use of HoloLens is not gaming," HoloLens general manager Scott Erickson said during the 2016 Build conference.
9. Microsoft is emphasizing productivity
The developers working on creating apps for the headset will have the ultimate say on what the HoloLens experience will be like, but two years of public demos have given us some flavor of what Microsoft thinks you’ll use its headset to do. At this year’s Build Conference, the dean of Case Western University's school of medicine demonstrated how the headset could be used in lectures with holographic representations of organs and body systems appearing in front of students.
Meanwhile, NASA and Microsoft showed off an app called Destination Mars that uses data collected by the Mars Curiosity rover to recreate the Martian surface; scientists with the space agency are able to explore Mars’ terrain without ever having to leave their office.
10. The HoloLens will change the way you watch video
It won’t be all work and no play for HoloLens. Prior to the Super Bowl, Microsoft posted a YouTube video showing the ways its headset could theoretically augment the way you watch football, from expanding the field of view beyond your TV screen to overlaying stats and game information on top of live action. We could see a future where HoloLens developers apply that approach to movies and TV shows, using the headset to summon up second-screen data about whatever program you’re watching.
11. Holograms will be interactive and collaborative
You won’t be the only person able to see those holographic images appearing on your HoloLens display. Microsoft’s developer tools allow app makers to create collaborative experiences in which multiple users will be able to see and interact with the same objects.
We got a first-hand look at this during Build 2016, during a hands-on session in which we had winged avatars hovering over our shoulders that other participants in the session could see. We were also able to toss objects at those avatars, with little cartoon stars appearing over their virtual heads when we scored a direct hit. It really drove home the point that when HoloLens ships, it doesn’t have to be a solitary experience.
MORE: HoloLens Hands-on
WHAT WE DON'T KNOW
1. When will HoloLens be available to consumers?
That remains unclear, as Microsoft hasn’t announced a commercial release date for either the software or the hardware. We also don’t know what the final price will be — we can’t imagine Microsoft will charge consumers the same $3,000 that developers are paying, but all signs point to the HoloLens costing more than virtual reality headsets from Oculus and HTC. In 2015, a Microsoft executive told the New York Times that the HoloLens would cost “significantly more than a game console.”
2. What does the general Windows Holographic interface look like?
We've seen Holographic apps, but we haven't seen the desktop and menus that HoloLens users will need to launch programs, switch between apps and change settings. Assorted demos at Microsoft’s Build conferences have shown a floating Start menu, but we still don't know what parts of the Windows 10 UI will be available in Windows Holographic. I'd love to see a taskbar, action center, app store and Start menu in Holographic form, but I don't know what Microsoft is doing to build out the OS for HoloLens users.
MORE: HoloLens Finally Makes Augmented Reality Real
3. Will Microsoft allow other vendors to make their own headsets?
Microsoft's operating systems, from MS-DOS to the latest Windows, have succeeded because so many different manufacturers are building computers for them. In recent years, Microsoft has built a number of its own systems, including the Surface line, Lumia phones and the Xbox One, but that hasn't stopped the Dells, HPs and Lenovos of the world from building Windows computers. I hope Microsoft will allow other vendors to build their own Windows Holographic visors, because competition like that leads to freedom of choice for consumers and lower prices.
4. How long is the battery life?
When we’ve used HoloLens during Microsoft-supervised coding sessions, we left the headset plugged into the computer to receive the latest code deployment. However, when testing apps, we were able to walk around rooms, with an untethered HoloLens. In its rundown of HoloLens specs, Windows Central says the headset has a 16.5Wh battery, which provides 2 hours of uptime. You would imagine that a commercial version of the headset would feature a longer-lasting battery to handle the kind of design, education and productivity uses Microsoft has outlined.
5. Who will make Holographic apps?
Microsoft has shown some of its apps running as holograms, including Skype and Windows Media Player. However, we don't know whether the major services we know and love on desktop/phone Windows will be available on the Windows Holographic platform.
How cool would it be if your Facebook wall could be placed on an actual wall? In order for that to happen, Mark Zuckerberg and company would have to spend their time and money developing a Holographic version of their popular social networking app. It has been difficult for Microsoft to get the major app developers on board with Universal apps and programs for Windows 10 Mobile — Google famously decided to snub Windows Phone — so we'll have to see whether big names like Netflix and Twitter are among the ranks of developers working on Holographic apps for HoloLens’ commercial launch.
Senior Editor Philip Michaels contributed to this report.
