Masters of Disruption: How the Gamer Generation Built the Future [31]
In 2017, I went to Microsoft to write about the HoloLens, and "mixed reality."
This post is part of a yearlong project I’m serializing exclusively in my newsletter, Disruptor. It’s a follow-up to my first book, Masters of Doom: How Two Guys Built an Empire and Transformed Pop Culture, and it’s called Masters of Disruption: How the Gamer Generation Built the Future. You can find the table of contents, as it unfolds, here. To follow along, please subscribe below. Thanks!
In 2017, I went to Microsoft to write about their “mixed reality” research and development. Here’s the unpublished longform story I wrote, which later appeared in shorter form in Rolling Stone.
The next big thing in tech goes back, as it often does, to Star Wars. You know that scene when Chewie and R2 are playing holographic chess? Instead of plastic pawns and rooks, the board teems with animated, three-dimensional alien, one of whom body slams the other like a Montana congressman. That’s mixed reality, nascent technology which lets you see and interact with holograms overlaid in the world around you.
To best view MR, you wear a special headset but a phone, a tablet, or any device with a camera will suffice. Yes, this is different than virtual reality, the other big thing. In VR, you strap on a big black headset so that you only see a computer-generated environment, like being inside a video game. While not as immersive, MR has one distinct advantage over VR: you can experience it as we do most media, without completely leaving reality behind.
The first taste of MR hit with Pokemon Go, the mobile game which uses a phone camera to populate little animated creatures into a real-world scavenger hunt. The novelty of seeing a red-horned, flame-tailed Charmeleon scurry over your sofa spawned a global phenomenon last summer, as phone-wielding humans with outstretched arms chased imaginary Pokemon into trees, oceans, and oncoming traffic. SnapChat filters, which use your phone camera to let you augment your face with bunny ears and Jagger lips, is another iteration of MR lite.
But with robust new technology in the pipeline, mixed reality (sometimes called augmented reality) is about to get way bigger, and weirder too. MR is already starting to transform daily life, from shopping to entertainment. At the National Museum in Singapore, visitors hold up their phones in front of a mummy to see an overlay of the busted skeleton inside. Lowe’s, the home improvement chain, has a mixed reality app that guides you through a store toward the items you want to buy; just type in, say, hex bolts, hold up your phone, and you’ll see a digital yellow line leading you down the right aisle.
With the MR market estimated to hit $133 billion by 2021, according to Zion Market Research, the leading names in technology are racing to bring the technology to market. Google has its mixed reality platform, Tango. Microsoft is pushing the HoloLens, a $3000 MR headset that so far is only available for developers. Then there’s the two main startups, Meta, a Silicon Valley company launched on Kickstarter, and Magic Leap, a secretive behemoth in Florida backed by the Chinese technology giant, Alibaba, among others, and worth an estimated $4.5 billion.
In April at Facebook’s annual developer’s conference, CEO Mark Zuckerberg called on programmers to create apps for its new mixed reality platform, Camera Effects. "Over time,” he said, “I think this will be a really important technology that changes how we use our phones.” While Apple has yet to specifically announce MR support, CEO Tim Cook predicts that MR will become so ubiquitous that “we will wonder…how we ever lived without it, like we wonder how we lived without our phones today.”
In real life, British beatboxing comedian Reggie Watts is about two hundred pounds and six feet tall, with an afro that adds another six inches. But the holographic version of Watts I see through my HoloLens headset is positively Lilliputian.
He stands, around a half foot high, on a white column inside a small, windowless studio on Microsoft’s campus in Redmond, Washington, where I’ve come to experience the technology firsthand. It’s a prerecorded performance, not yet commercially available, and a compelling example of how MR changes the way we can experience the world around us and each other.
Dressed in bright red pants, a gray t-shirt, and gray suspenders, HoloWatts beatboxes an alien tune as he twiddles the knobs on a tiny red sampler. He looks pretty real, consider. But since this is computer-generated, the extent of his realness is malleable. In this performance, his “stage” resembles like a Yes album cover come to life, a crystalline blue cove with neon pink water and an undulating green orb cradled in incandescent blue vines.
Because the HoloLens is wireless, I can view Watts from any angle in three-dimensions as I walk around room. While he’s rendered on a small scale now, he could just as easily be life-size – or 100 feet tall. Plus, because he’s a software program, he could be downloaded and viewed anywhere, anytime. In the near future, instead of schlepping to Giant Stadium to see U2 in the flesh, you can watch a full, lifelike, 3D concert in your living room.
To get a better idea of how such experiences are created, I’m given the first behind-the-scenes tour of Microsoft’s hologram factory, which they call the Mixed Reality Studio. Steve Sullivan, the former senior technology officer at George Lucas’s Industrial Light and Magic effects studio heads up this one. Inside the large warehouse, dozens of cameras crane down to “capture” someone and turn them into holograms.
Watts, George Takei, and dancers for Cirque Du Soleil are among those who’ve been captured so far, and when I get digitized, it takes in a matter of minutes. The plan is to have Mixed Reality Studios in every major city around the country, and the first will roll out in San Francisco later this year. Think of it like a 21st century recording studio, soundstage, and photo booth all rolled into one. Bands, educators, families, and so will drop in, get captured, and then be available for download. As Sullivan tells me, “the point is to get this out to the world.”
In another studio, I try out a mixed reality shoot ‘em up game called RoboRaid. Donning the HoloLens, which creates a real-time map of the room, I watch as teams of spidery robotic drones burst from the walls and fly toward me. To shoot them, I extend my forefinger and flick it down to fire off lasers. With each blast, a simulated gaping hole appears in the wall, exposing the wooden beams and pipes behind it. Though it feels goofy to battle by finger flick, the experience of gaming on two feet, freed from a controller or TV set is transporting. And, unlike VR, I can still see the environment around me, so I’m not hung up about tripping over a coffee table or checking a text on my phone.
But beyond entertainment, MR can radically change how we socialize and learn. In another studio, I’m given a demo in which a woman in a remote location teaches me how to install a light switch. With my HoloLens on, a Skype window opens up before me as if it’s floating in space. Because the caller can see what I see, she’s able to guide me along step by step as I tighten wires and reach for tools. She can even annotate what I’m seeing – drawing a circle around the screw I’m supposed to tighten. “Good job,” she tells me, as the overhead light comes on.
At Case Western Reserve University in Cleveland, radiology professor Mark Griswold is teaching anatomy on holographic cadavers instead of the real things. Rather than being stuck behind desks flipping through textbooks or even webpages, students gather and roam as a hologram of an aortic valve floats before them. By 2019, the University will replace its cadaver lab with a mixed reality classroom. “Were upending several hundred years of how we teach medical students,” Griswold says, “we can explore the human body better than we ever could in the past.”
“Did you walk on Mars today?” Alex Kipman, the visionary inventor of the HoloLens, asks me in a sun-dappled Microsoft conference room.
He’s referring to a mixed reality experience that transforms your room into a virtual Red Planet. The project was developed over seven years with NASA’s Jet Propulsion Lab, and uses data and images collected by the Curiosity rover to enable scientists to explore and collaborate in ways an ordinary computer doesn’t allow. Instead of pointing and clicking at a photo of a rock on a flat computer screen, geologists see it in three-dimensions in the context of the larger landscape. Because it’s social, several researchers around the world can annotate the experience in real-time – with the kind of “hey look over here” interactions that come with wandering a landscape together.
“We want to give you the ability to just deal with technology over the real world, as opposed to dealing with technology behind a digital screen,” Kipman says.
It’s fitting that Kipman, a long-haired 36-year-old Brazilian in ribbed jeans and a military jacket, seems to model himself on Criss Angel. When he describes how the HoloLens lets “you walk around the world with a reality mixer,” he sounds more like a magician than an engineer. It’s not the technology which turns him on, it’s the power of the illusions to foster new kinds of human experience. “The vision is not about holograms or head mounted displays, those are just lenses,” he tells me, “They're making the invisible visible. What is the invisible that they're making visible? People, places, and things.”
For Kipman, the vision has been a long time coming. As the middle son of a traveling diplomat, Kipman grew up in cities around the world, where his only constant was his Atari 2600 and his computer. Instead of investing in new friendships, he devoted himself to programming. “There was escapism for me in physics and math,” he recalls.
For engineers like him, the ultimate escape has long been imagined as virtual reality. But while programmers were stymied by slow processing speeds and unwieldy headsets, Kipman was taking a counterintuitive approach. Rather than trying to solve the problem of making people convincingly perceive a computer world, he focused on empowering computers to perceive us. The better they do, the more seamless our experiences can be – so that, instead of, say, pressing a button on a controller, someone can just flick his finger. “That’s why we have all of this machine intelligence,” he says, “there’s no set-up, there’s no sensors in the environment, stuff just works.”
This pursuit led to his first breakthrough invention: Kinect, the motion-sensing camera for the Xbox 360, which, upon its release in 2010, became the fastest-selling consumer device of all time. For Kipman, it was the realization of his dream of “a machine observing the world,” as he says, and the basis of what would become the HoloLens and the mixed reality of today.
But for all the magic, the computer-generated dream is hitting up against realities of its own. Facebook’s VR company Oculus has been embroiled in legal and technical challenges, relegating it to the back of the pack behind Sony’s PlayStation VR headset and the HTC Vive. Magic Leap, the company which generated most of the mixed reality headlines after its launch in 2010, has been recently dogged by rumors of delays and a chunky headset that detracts from the experience. While Google is busy rolling out Tango [the program was shut down in March 2018], it’s still reeling from the crash of Glass, the mixed reality headsets it discontinued in 2015 after only two years in production.
Part of the problem is that people have yet to take to wearing computers on their faces – especially when they look more like Daft Punk accessories than Ray-Bans. But Jonny Lee, director of engineering for Tango, Google’s MR platform, says the fantasy of wearing computerized contacts instead is just that, due to the needs for more robust hardware and battery power. “We’ll likely always need something we have to wear,” he says.
While a variety of MR headsets are hitting shelves this year, including the first consumer version of the HoloLens expected this holiday season, many of the mixed reality experiences will continue to be available for computers and phones. “We’re starting with phones,” says Lee, “because that’s what most people have.” For now, the focus among developers is less about how we experience mixed reality, than on making it real for everyone. As Kipman says, “It’s much harder to create something timeless than it is to create something fashionable.”
