What Apple’s Patents In AR, AI, And Facial Recognition Mean For Future iPhones

With Apple’s release of the iPhone 8 line and the high-end iPhone X (which commemorates the iPhone’s 10th anniversary), we have a new vision of where we are in personal technology today. But the release is only what’s fit to ship for this coming fall.

To gain insight on where the future of iPhones could be headed, we looked through Apple’s latest patents in bleeding edge computing technology like computer vision, advanced AR/VR, and advanced materials.

While certain technologies like augmented reality (AR) are an expected focus in near-term iPhones, other technologies — like futuristic hologram displays — are more surprising, and may be several years out. All the patents listed blow were surfaced using the CB Insights patent search engine.

1. Holograms And 3D displays

The above patent application, called “Interactive three-dimensional display system,” details a system that displays images that can be manipulated by users in mid-air, without the need for special glasses. The application, which was filed in October 2012, mentions the fields of education, medical diagnostics, and biomedical engineering as possible application areas.

2. Headset VR

While it may be years away, Apple has a patent for a head-mounted display (HMD) that borrows from the popular mobile HMD design that slides in an iPhone as a VR display. Apple’s vision for their HMD would also cater to its device ecosystem, with handheld remote controls that would connect to the headset.

Below is a drawing from “Head-mounted display apparatus for retaining a portable electronic device with display,” which was granted in May 2017.

3. Flexible OLED displays

The new iPhone X employs organic light emitting diode (OLED), a display that extends edge-to-edge. OLED’s superior contrast ratio, color, and power efficiency makes it the obvious evolution from the LCD screen, and will become ubiquitous.

Apple’s reliance on OLED suppliers may make early iterations more expensive, but Apple has applied for more than 20 OLED-related patents since 2009, and will likely continue to focus on this tech for future phones.

This January 2017 application, “Flexible electronic devices,” envisions super-flexible iPhones and mentions in the patent language it could be for both LCD and OLED type screens.

4. Digital scrolls

In a patent that was granted in April 2016 called “Electronic devices with retractable displays,” Apple opened the possibility of e-paper being a focus in future iPhones. Perhaps piggy-backing on its IP for flexible OLED, the patent details a future device with retractable ends that would open a large scroll-like screen:

5. Iris recognition

With the next wave of smartphones incorporating facial recognition, Apple has other biometric logins in the works. Its patent for “Using biometric verification to grant access to redacted content” was granted in July 2017, and mentions iris-reading as a potential unique identifier:

“The biometric reader reads a person’s uniquely identifying biometric data (e.g., thumbprint/fingerprint, iris scan, voice, etc.). … When the biometric data matches the stored data, the person is verified.”

The patent’s images show how personal information in a text message could be blurred out until the user is verified by a biometric marker:

6. Machine Learning Chips

To handle all the computation from budding technologies like AR and Siri’s natural language processing (NLP), Apple was said to be developing a specialized AI processor for the iPhone. Interestingly, the company only has 11 filings with mentions of “machine learning” or “neural network,” even though it introduced its Core ML machine learning library in June.

Apple’s latest relevant patent application is from May 2016, and lends some clues that speech-to-text and Siri’s NLP will be a focus of Apple’s AI efforts. Titled “Applying neural network language models to weighted finite state transducers for automatic speech recognition,” the patent details a neural network process for NLP as seen below.

Indeed, at today’s event, Apple announced the high-end model iPhone X will ship with the AI chip that many observers thought might not be ready in time. Called the A11 Bionic neural engine chip, the processor is custom built for AI workloads and will presumably be employing its patented neural networks and processes.

7. facial RECOGNITION

The new iPhone X will feature brand new facial recognition technology to supersede the fingerprint-based TouchID. But the release will like take some time to perfect — as TouchID did — as the feature is updated to improve fidelity and speed. Unsurprisingly, Apple has secured 5 grants and 2 outstanding applications patents in facial recognition.

Here’s a drawing from the recently granted “Locking and unlocking a mobile device using facial recognition,” which could be a key differentiator for future iPhones.

Branded as FaceID, Apple revealed that the iPhone X will use this facial recognition technology for unlocking phones, as well as making purchases with ApplePay. The company stressed its system’s ability to recognize accurately in spite of facial grooming, hair styles, and natural aging, and will prevent unauthorized entry with photographs and masks.

Also notable is the timing of Apple’s acquisition of RealFace, an Israel-based AI company specializing in facial recognition. Apple acquired the company in February 2017, as we detailed in our M&A history of the tech giant.

8. iphone Stylus

The late Steve Jobs was famously not fond of the stylus, and instead helped usher in the now-ubiquitous multi-touch mobile phone screen. Despite the historical distaste for styluses (not to mention the failed Newton PDA in the late 90s), Apple has recently released the Apple Pencil to work with the iPad Pro display, and has secured several patents for its electronic stylus.

Apple was granted this simple stylus design in June 2017:

But more recent applications, such as this September 2016 “Stylus for electronic devices,” has a more connected vision that would include a power or a USB connector at the end:

9. AUGMENTED REALITY Maps

With a big push for AR coming from the top, Apple has not minced words on its desire to be a dominant player in augmented reality. Apple currently has 3 applications and 6 patent grants in augmented reality. AR apps are expected to be a big part of the push around the new iPhone 8 and the iOS 11 operating system.

Inside a patent application granted in November 2016, Apple’s near-term vision for augmented reality comes into focus. The patent grant, titled “Augmented reality maps,” shows how the mobile AR could easily be overlaid onto real-world scenery.

10. Mobile AR

A patent titled “Synchronized, interactive augmented reality displays for multifunction devices” that was granted in 2013 envisions what we’ve seen with many of the ARKit demos coming out, and allows users to film a live scene and get an overlay. The language describes that “a frame of live video can be ‘snapped’ (e.g. by pressing button) and processed using known object recognition technique.”

At today’s release, Apple’s latest batch of phones were designed ground-up for augmented reality, with factory calibration for AR, along with new gyroscopes and accelerometers to ensure accurate tracking. The game studios behind the first AR titles like Warhammer and The Machines could be working atop these early Apple patents.