A look at the current state of virtual, augmented, and mixed reality — and why these technologies will be the building blocks for smartphone disruption.
Smartphones are now hyper-personalized. With some 6 billion apps available on iPhones and Androids, smartphones serve as everything from email machines to AR-based Pokemon-catchers.
But their small size isn’t well suited to the myriad tasks they now perform. And simply increasing screen size doesn’t really resolve this problem — especially as people increasingly look to their devices for immersive media experiences.
Devices sales have also reached saturation. Today, growth in smartphone shipments is effectively negative, and smartphones are primarily bought as replacement products.
So what’s next for the smartphone?
Mixed reality (MR) offers a new digital experience that isn’t confined to handheld mobile screens. The tech — also known as hybrid reality — refers to the merging of real and virtual worlds.
And it’s becoming an increasingly mainstream concept: there’s been a substantial uptick in media coverage of MR since 2017.
Immersive headset technology and voice assistant software are beginning to allow users to experience new environments and visualizations, where physical and digital objects interact in real time. With this tech on the rise, MR is poised to become the next major computing platform.
Below, we analyze the VR, AR, and MR technologies that are the building blocks for disruption in the smartphone industry.
After the smartphone
Digital media usage continues to rise, now reaching almost six hours per day, with smartphones driving much up the uptick.
The smartphone — and its screen element in particular — has become the entry point to the online world.
As content becomes richer and bandwidth speeds increase, consumers will look to move beyond smartphone screens for more enhanced experiences.
Unbundling the smartphone
Wearables represent the beginning of smartphone unbundling, as many smartphone services and apps become accessible via new interfaces that don’t have to be tucked away in a pocket or handbag between uses.
Wearables have paved the way for a profound tech transformation in industries spanning apparel, fitness, and healthcare, with notable companies including Fitbit (IPO in 2015), Withings (acquired by Nokia for $191M in 2016), and Misfit (acquired by Fossil for $260M in late 2015).
While Fitbit has seen its stock price decline in recent years and wearables are still far from ubiquitous, there’s no question that wearable devices like smart watches are becoming much more popular.
Tech giants are helping push the category: Apple has jumped to the forefront of the smart watch category, while Snap has been working on its Snapchat Spectacles.
Snap’s augmented reality and emotion recognition patents, which focus heavily on its Spectacles, offer insights into what the next mobile computing platforms might look like. The company stealthily acquired eyeglass video camera developer Vergence Labs at the end of 2014.
Patent images for augmented reality smart glasses below.
The patents depict a future where digital images — like a rampaging dinosaur — can be superimposed on top of real settings, like a city street.
The digital world is no longer confined to a screen
As consumers seek out more immersive experiences, it becomes increasingly important to understand the difference between virtual, augmented, and mixed reality, which will be the backbone of the shift away from traditional screens:
- Virtual reality (VR) is fully immersive, and places users in a completely digital environment.
- Augmented reality (AR) overlays digital elements onto a real-world setting (e.g. Pokemon Go).
- Mixed reality (MR) is a combination of the two, allowing the user to interact with physical and digital elements simultaneously.
Virtual reality is perhaps most similar to the way we consume much digital media today, in that users must disconnect from physical reality.
Mixed reality can be thought of as a more advanced AR technology, and could see faster adoption because it will be experienced as a consistent overlay in everyday life. It could become one of the dominant technologies for daily living and the key experiential element of the next mobile computing platform.
The VR/AR market landscape
The AR/VR market is estimated to have enormous growth potential.
The market is projected to reach $80B by 2025, according to Goldman Sachs estimates. (Broken down further, the firm pegs total addressable market for AR/VR software at $35B, and hardware at $45B.) Meanwhile Citigroup has estimated the sector will grow to more than $2T by 2035, as more industries and applications take advantage of the tech.
AR alone is a large and growing market.
Driven by Pokemon Go’s outsized success, AR grew to an unexpected $1.2B revenue in 2016, helping bolster the overall VR/AR market to $3.9B that year.
Entertainment, media, and content (specifically gaming) are the most obvious industries for AR/VR use cases. But new applications in industries like healthcare, education, military defense, and real estate are appearing every day.
Currently, there are many devices — head-mounted displays (HMDs), smart glasses, phones and tablets, head-up displays (HUDs) — that support AR, and some of these will likely become the backbone for mixed reality platforms. Recently, Florida-based unicorn Magic Leap released the “creator edition” of its Magic Leap One, an AR headset that allows 3D objects to interact with the physical world.
Other pioneers in the space include Microsoft’s HoloLens, a self-contained holographic computer and head-mounted display that allows users to interact with holograms in the real world, immersive AR headset Meta, and a forthcoming AR/VR device from Apple, set to release in 2020.
Developer tools are also growing. Apple’s ARKit, launched in summer 2017, can use the iPhone’s camera sensor to estimate light and apply shading and texture to virtual objects, while Vuforia offers an AR mobile software development kit that uses computer vision to recognize and track images and 3D objects. ARToolKit (first released by the University of Washington HIT lab nearly 20 years ago) offers an open-source computer tracking library for creating AR apps,
Other notable developer platforms include Unity, Wikitude, Google’s ARCore, and Facebook’s AR Studio.
The tech behind the rise of MR
Mixed reality is the next big paradigm shift when it comes to human-computer interactions. While virtual reality shuts out the surrounding world, and augmented reality imposes a virtual layer on top of it, mixed reality offers a truly integrated digital-physical experience, where the real and virtual worlds can more fully interact.
To get to this integrated state, MR will rely on technologies like voice recognition and predictive AI to better observe, interpret, and respond to user behavior.
leveraging voice for an immersive user experience
Mixed reality goes beyond VR/AR to more seamlessly blend the digital and real worlds.
In order to do so, the tech will rely on smart goggles hardware, as well as on virtual assistants with advanced speech recognition for a complete end user experience.
Voice recognition is becoming a critical technology for the current and next generation of smart devices, and a key interface in new MR platforms. Commercial usage is spreading in the industry, including media news transcription, voice search, and real-time speech translation.
Executives are also taking note: mentions of “voice recognition” on earnings calls have increased in recent years.
Voice-based user interfaces are becoming increasingly popular, and changing the way people expect to interact with computers. Assistant tools like Amazon’s Alexa, Google’s Assistant, Microsoft’s Cortana, and Apple’s Siri are gaining wide adoption, leading to less reliance on the screen.
Today, nearly 1 in 5 US adults has access to a smart speaker, with adoption of voice-powered devices growing to 47M in just two years. (For reference, it took 13 years for televisions to reach the 50M mark, and 4 years for internet access to reach 50M.)
Google’s Duplex is one example of how far voice assistant technology has come. The voice system, which began public testing this summer, can carry out phone calls on a user’s behalf, such as checking business hours or booking a restaurant reservation.
Google’s voice-activated speaker Home powered by the Google Assistant was launched in 2017, and last year saw the integration of Google Assistant into TVs. According to Google, the Assistant is now available on more than 400 million devices.
As consumers grow accustomed to computing by speech, voice technology will lay the groundwork for the hands-free software that will underpin mixed reality.
using ai to predict user intent
Over the last 60 years, as computers have become more powerful, human interactions with computers have evolved from punch cards and keyboard inputs to using natural speech and gestures, with ongoing research into thought-controlled computing.
This represents a broader shift from explicit interaction (the end-user instructs the computer what to do) to implicit understanding (the computer observes end-user behavior and implicitly infers, predicts, and responds to user intent).
Microsoft’s book The Future Computed describes a normal day in the year 2038, highlighting the ways that technology will affect daily life.
Even before the hypothetical day begins, tech assistants are hard at work optimizing a user’s day-to-day:
“Your digital assistant goes through your calendar and talks to your other electronic devices to plan your day while you sleep.
With an eye on your sleep cycles, it wakes you at a time in which you will feel most refreshed, within a window of time you’ve previously approved.”
Here, the action of activating an alarm is built on the user’s intention to maximize their sleep cycle.
The book details the other ways tech will influence daily life, including cars that select optimal travel routes in order to minimize commute time and virtual assistants that anticipate shopping orders based on recurring events like birthdays.
It also highlights how mixed reality can be used to manage virtual meetings, responding to user intention to optimize effectiveness and flow and itemize follow-up tasks:
“Your first meeting for the day will be with an international team and held remotely. Before you leave for the office, you put on a pair of mixed reality glasses and greet your colleagues, who appear before you in a virtual boardroom. You all put in an earpiece so that each side’s language is automatically translated for the other, without lag.
After the meeting, your assistant shows it has prepared a summary of the discussion and has determined a task list based on next steps.”
The list of tech use cases in just a single day goes on and on, with an underlying pattern of predicting user intent.
The goal is to have sufficiently advanced technology that can accurately predict what users want, while learning continuously from end-user behavior, acting or suggesting next action to the user in an anticipatory fashion (as our predictive keyboards do when we type in our smartphones).
As VR and interactive 3D developer David Smith has pointed out, traditional mouse & keyboard tech has been widely successful because of user intention amplification, where small movements and gestures result in large, significant actions. In the MR space, there is additional potential to allow users’ small actions — like looking at a certain object or making a certain gesture — to have a large effect on manipulating devices.
By centering design on a user’s intentions and using intelligent objects (virtual and otherwise) that respond to that intent, developers allow users to bypass physical limitations, enabling a whole new level of freedom.
Key players to watch
An OS war is coming
The MR platform will be made possible by a number of converging technologies, including head mounted displays (HMD), AR/VR tech, artificial intelligence, and cloud-based services. Several major tech companies are already offering partial or early implementations of it.
In June 2017, Apple introduced its AR toolbox ARKit (mentioned above). In October 2017, Microsoft announced new “mixed reality” features for its Windows 10 Creators Update, including its first handheld MR motion controllers, upgrades to its MR headset, and the ability to add 3D objects to photos and videos.
Operating systems and platform services are already evolving to include features for VR/AR and MR technologies: Windows Mixed Reality Platform APIs became a component of the Universal Windows Platform and Android Nougat introduced support and optimizations for virtual reality.
Google’s involvement in MR goes far beyond current OS updates: in 2014 Google led a $542M Series B to MR HMD manufacturer Magic Leap, and it participated in a $794M follow-on round to the company in 2016.
Magic Leap announced its first commercial MR smart glasses in December 2017, followed by the launch of its SDK (software development kit) cloud tools in March 2018. Its Magic Leap One headset had a limited release earlier this summer.
With these developments, the race for MR is already underway.
Who’s leading the way?
A number of established computing players are heavily betting on VR/AR, with many focusing strategically on MR and hybrid reality.
Google, Apple, Microsoft, Facebook, and Snap have all positioned themselves in this new market, and some frontrunners are already pulling ahead.
Right now, two main MR goggle product platforms are emerging: the Microsoft HoloLens and Magic Leap One.
HoloLens
Microsoft has undertaken a massive effort in the world of MR computing with its HoloLens headset. The product features a number of sensors and cameras that scan the world around the end user to create a 3D mesh rendering of their surroundings. This “spatial map” then enables digital image overlays.
This spatial map also allows “occlusion,” or the hiding of virtual objects behind real things. This allows digital images to interact in 3D with real life objects, moving around or hiding behind them (rather than just being superimposed in front of everything).
The main interfaces for the HoloLens are gesture and voice control, though the device also allows legacy device connection through Bluetooth (so a user can still use a keyboard, mouse, game controller, etc.).
Hololens hardware has three processors: a CPU and a GPU, as well as a custom processor designed to handle the sensor array input called the HPU (Holographic Processing Unit). The headset contains 2 GB of RAM and 64 GB of storage.
In addition to the device itself, Microsoft has been building an ecosystem called Windows Mixed Reality. This special version of Windows 10 is designed to combine VR and AR experiences with the Windows 10 operating system, along with a growing array of MR applications made available through Windows Store.
Having missed out on the smartphone OS wars, Microsoft may be more intensely focused on aggressively competing in the next battle for OS market share.
Magic Leap
Magic Leap One (Creator Edition) is the Magic Leap’s first commercially available product. The headset is tethered with a cable that connects the HMD to a small hip-mounted computer called Lightpack, which handles graphics processing.
Another smaller computer in the goggles handles sensors and world perception and includes machine learning capabilities. It has four built-in microphones, six external cameras, and built-in speakers.
With this, and the proprietary development of light fields for near-eye optics, this headset is significantly smaller than the HoloLens and it features better specs in terms of depth of field and FOV. The ability to blur and focus virtual objects enhances the experience within our real world when using the device.
How will they stack up?
The HoloLens and Magic Leap One are comparable products, and are similar in terms of camera, microphone, and speaker technology. While the HoloLens’ field of vision (FOV) is smaller than that of the Magic Leap One, Microsoft’s focus on occlusion may help it distinguish its experiences as the technology evolves.
As the two compete for market share, key differentiators will be not only how immersive and successful each device is at creating a reimagined and realistic world, but how easy it is for users to forget they’re wearing the technology at all.
Just as the sleek design of the iPhone played a key part in its success, the design of hardware worn on the face will likely come down to how comfortable and unobtrusive headsets can be.
Conclusion: is MR a sure bet?
The fundamental technologies for a new mobile computing platform to emerge and replace our smartphones are already here.
Building off of AR/VR technologies, mixed reality has the potential to transform the way consumers experience content. Your next pair of glasses could correct your vision and act as an eyewear computer to replace your smartphone.
Similar to what we saw in the early days of the PC industry, and later on in the mobile industry, we will likely see a wave of a few early MR smart glasses products for consumers launching by the end of this year. Magic Leap is already off to the race, and HoloLens is on track to join later this year.
While some early movers may fail, the huge advantages of a computing surface that isn’t restricted by a phone or computer screen and that can dispense with typing to input commands makes mixed reality an extremely compelling proposition.
In the next decade or so, mixed reality is expected to overtake current VR/AR markets to become the dominant technology for everyday computing. MR will rely on smart goggles and headsets as its hardware platform, combining virtual assistants with advanced speech recognition and anticipatory AI for a seamless and immersive user experience.
The combination of these three drivers — MR technology, voice driven virtual assistants, and anticipatory AI — will ultimately drive the adoption of new mobile computing platforms, expanding the mobile experience off our screens and integrating it more completely into the world around us.
If you aren’t already a client, sign up for a free trial to learn more about our platform.