Snap Inc. — formerly known as Snapchat — held its first quarterly earnings call with investors today. The company, which went public at a $24.8B valuation earlier this year, faces user growth challenges along with intense competition from Facebook’s Instagram. On the call, analysts asked about Snap’s plans for augmented reality (an arena in which Facebook has also said it is expanding into), as well as the performance of Spectacles, its camera-laden sunglasses for capturing video.
Snap’s executives did not reveal much, only that they would continue to develop bleeding-edge tech internally. But studying the patent applications and grants of the Santa Monica-based “camera company,” you get forward-looking insight into its product roadmap and innovation strategies.
Recently published or granted patents reveal possible areas of future product direction for Snap, including AR, facial recognition, and ad-monetized video calls.
What Snap’s working on
1. Augmented reality
Inside a patent application published on May 4 2017, Snap’s augmented reality ambitions became much clearer. Interestingly, it’s the first of Snap’s patent efforts that use the specific phrase “augmented reality.” The patent application (#20170124713), which is titled “Image Based Tracking in Augmented Reality Systems,” details technology that could map onto surroundings and superimpose life-like moving images such as a Tyrannosaurus Rex.
The patent also invites the possibility of this being a headset-based AR experience, and included drawings of incorporating what appears to be future iterations of its Spectacles.
2. Emotion recognition
A patent granted in February 2017 titled “Emotion recognition in video conferencing” (#9576190, granted 2/17/17) details a computer vision system that would use a “virtual mesh,” much like its existing Snap lenses, to detect facial features and identify reactions.
Curiously, the patent says it’s for two-way videoconferencing, as opposed to Snap’s famous ephemeral messaging. Also interesting is that the patent suggests Customer Support could be a major commercial use case:
Although videoconferencing creates a useful channel to provide services to customers, one challenging issue is to deal with irate, annoyed and distressed customers. Customer anger is not always easy to spot even to professional service providers or sales representatives. One of the important strategies in sales and customer service is to detect when a customer is angry or annoyed and prevent difficult situations at a very early stage. Unfortunately, it is a very difficult task even to experienced professionals.
3. Video calls with commerce/payments
One notable patent grant (#9083770, granted 7/14/2015) indicates plans for a “real time communications” video streaming platform that could connect people, much like Skype or FaceTime, but also incorporates a payments platform. The patent list 4 inventors, all of whom were engineers or co-founders of AddLive, which Snap acquired in April 2014.
(The below suggest a video call with an attorney that charges $50.)
The patent mentions low-latency audio as a focus, and gives some example graphical interfaces (below) that introduce concepts of a payment platform bolted on to this “real-time communications” feature. Snap has experimented with a peer-to-peer payments platform known as Snapcash, and already provides video calls in the style of FaceTime or Skype. But doesn’t currently have a built-in payment or commerce mechanism on the platform.
The patent goes into the system languages, APIs, and packet flow for this real-time communication (RTC) tool, and goes on to describe the commercial applications:
In some embodiment, the RTC application can also be used for commercial purposes. For example, service providers, such as attorneys, can providing [sic] counseling to their clients using the RTC application. The service providers and the client may view and hear each other using the RTC application.
4. Audio/acoustic fingerprinting
As a way to manage galleries of Snap videos taken at live events, Snap has applied for a patent (#20160182875) on its audio fingerprinting process. The patent was published in June 2016 but filed for late in 2015. Acoustic fingerprinting is a common method for copyright compliance, licensing, and monetization. In this process, computers can scan video and audio for certain audio clues (inaudible to human ears) that effectively serve as a flag to mark certain content (e.g. content that is owned by a certain copyright holder).
With these audio fingerprints, videos of a live event could be grouped together in their proper order based on audio analysis to see if there is match, or perhaps to prevent showing redundant clips of the same moment:
The audio fingerprints are matched to known audio fingerprints to establish matched audio fingerprints. A determination is made whether the matched audio fingerprints correspond to a designated gallery constructed to receive a sequence of videos set to an audio time line. The matched audio fingerprints and corresponding video content are added to the audio time line. The operations are repeated until the audio time line is populated with corresponding video content to form a completed gallery with video segments set to audio segments that constitute a complete audio time line.
Conceivably, this means a process figuring out if a stream of Snap Live Stories (video and photo streams tied to events) are happening concurrently at a given concert or event, in order to automate the process of building the Stories that Snap curates and broadcasts as a Live Story to its users. For example, it may offer a Live Story of images and videos from a popular music festival.
But acoustic fingerprint technology could also suggest a monetization scheme for clips and copyrighted content in a manner similar to YouTube’s “ContentID” system for flagging copyrighted material and making sure revenue produced by advertising against that content goes to the copyright owner (e.g., a music label or movie studio).
5. Object recognition
In order to categorize Snaps (ephemeral photos and videos) and Snap Stories (photos and videos that last for 24 hours) into funnels like music festival-themed event-tied videos or city-based streams, Snap is analyzing the contents of messages. Snap has also applied for patents in object recognition, with an application that suggests tailored photo filters depending on objects contained in the video message. This patent was applied for in January 2015 but published in July 2016.
Snap has also imagined the technology having a possible commercial application with advertisers, as imagined here with coffee brands, which could be a possible direction for sponsored content (e.g. a brand could pay Snap to include filters tied to its brand that trigger when a relevant certain object is detected):
6. Facial recognition & privacy protection
Granted just in July of 2016, Snap designed a method (#9396354) for running facial recognition on images and then applying privacy rules to the image according to user preferences. Simply put, Snap could amass a database of faces and blur out faces in public video/photo messages for those users who have opted for privacy in their settings.
The patent language cites ephemeral galleries (Live Stories) as possible applications of the process:
A method executed by a computer, comprising: receiving an image from a client device; executing a facial recognition technique against an individual face within the image to obtain a recognized face; applying privacy rules to the image, wherein the privacy rules are associated with privacy settings for a user associated with the recognized face; and distributing a privacy protected version of the image, wherein the privacy protected version of the image has an altered image feature, wherein the privacy protected version of the image is distributed to an ephemeral gallery
Basically, if you’re a private opted-out user who find their way into a Snap story that gets picked up by the Live Stories stream (i.e. someone at a music festival films you in the crowd), the event stream would blur out or otherwise stop the image from being in the public broadcast.
The technology is in a similar vein to Facebook’s DeepFace, a facial recognition technology for auto-tagging friends in photos with 97.3% accuracy (you can opt out of this). Facebook is currently facing lawsuits for its handling of biometric data with DeepFace, so Snap would need to tread cautiously in this area.
Snap’s M&A for patents
Aside from the streaming payments platform mentioned above (that was ostensibly a product of the AddLive acquisition), we rounded up a few more patents originating from Snap’s M&A. Snap has been one of the most aggressive companies in terms of acquiring other startups, with 10 such acquisitions to date. Patent data reveals how M&A can help a company build its product strategy by acquiring outside innovation.
1. Recognition technology for 3D selfies
Snap quietly bought Seene in June 2016, and Seene’s parent company Obvious Engineering applied (#20160284123) for patents covering 3D facial recognition that also have an AR/VR application. Combined with Looksery‘s selfie filters (an image layer of fun special effects that is placed atop the original image), which Snap bought in September 2015, Seene’s software could add to the depth and flexibility of Snap’s arsenal of selfie filters. More accurate face mapping and 3D imaging would allow selfies and pictures to be taken in 3D (meaning, for example, that a three-dimensional selfie could be used to create a cartoon-like image of a Snap user).
Check out the patent drawings for its mobile app, featuring room capture & recreation and facial mapping (featuring a bearded user):
Seene’s product allows you to capture 3D objects (such as a statue or sneaker) and explore them within your smartphone.
2. ‘Eyewear having linkage assembly,’ a.k.a. Spectacles
Snap has two major filings related to its Spectacles eyewear, and both patents (which were granted November 1st 2016) list former Vergence Labs engineer David Meisenholder as an inventor. Vergence Labs was stealthily bought by Snap for $11M in cash and $4M in stock in December 2014.
Vergence Labs’ designs for camera laden glasses were fashionable and, interestingly, its camera housing (right) was very inconspicuous compared to Snap’s final Spectacles design (left).
Snap’s patents describe the special in-case charging mechanism on the Spectacles that latches to the eyewear in a manner similar to Apple’s magnetic charger on the MacBook. The design was incorporated into the Spectacles case.
And the patent drawing details exactly where on the frame hinges the charging ports are available:
Existing tech
While these patents have already been incorporated into the app, they still shed light on how the company has innovated in automation, security, privacy, and user experience/personalization.
1. Machine-driven content curation
Snap filed for IP related to automating the curation of Events-related galleries, which the company calls Live Stories (and was granted patent #9385983 in July 2016).
Where traditional media needed a centralized editing room to parse and piece together disparate camera angles and source into a coherent narrative, Snap has laid down IP for its vision for automating this curation process (likely working in its parsing of images and video with tech like object, facial, and audio recognition).
The language of the patent says:
A determination is made that the message parameter corresponds to a selected gallery, where the selected gallery includes a sequence of photographs or videos. The message is posted to the selected gallery in response to the determination. The selected gallery is supplied in response to a request.
The patent drawings show a familiar UI that suggests Event galleries for users to select.
Here’s a look at the patent specifics:
2. Internet Protocol-based intrusion detection
Granted in September 2015, the company’s patent (#9148424) relates to a method for preventing logins from unauthorized users. The patent describes a method to analyze the IP address of computers accessing a certain account to determine if it links up with the history of the authorized user.
The language describes an analysis method to prevent malicious account takeovers:
A login history comprising login request data for the server computer is analyzed to identify a plurality of usernames, wherein each username of the plurality of usernames is associated with a corresponding login request from the first IP address within a threshold time period of the first request time. In response to determining a login success ratio is below a threshold login success ratio and a number of unique usernames in the analyzed data is above the unique username threshold, the system automatically performs a security action.
3. Content delivery network for ephemeral objects
Snap’s secret sauce in the early days was its “self-destructing” messages. Unsurprisingly, the company sought several patents around its ephemeral messaging technology.
Grant #8914752, which was first applied for in 2013, gives a pretty simple schematic:
4. Social network profile access
Snap was granted a patent (#9026943) in May 2015, so that user settings could dictate how much profile information they give out to others. The patent states that the idea comes from concerns about social networks having “hard to find” privacy settings that make it difficult to know what’s visible to other users:
And the patent describes a method for implementing a way for users to manipulate their settings in one touch:
5. Permanent and semi-permanent messages
Snap applied for patents in 2014 that suggest something similar to what became the photo-management module known as Snapchat Memories, which allows users to browse photos and stories at a later time.
In part the patents focus on messaging replay (a feature that still exists, allowing a 1-time replay of disappearing messages). But they also break from Snap’s famous ephemeral messaging and allows users to save the content, showing the team at Snapchat was thinking about different types of permanence early on. Opting to have certain messages stay permanent could encompass message replays, but also Stories, and Snap’s latest big focus with Memories.
6. Hybridization of voice notes and calling
Snap got into the voice-calling realm with a late 2015 application for a calling and voice-messaging hybrid (#20160147435). Although it’s not a well-known feature, voice & video calling has been in the app for some time.
7. Ghost barcodes
In February 2016, Snap was granted a patent for its process with its ghost barcode reading (#9111164), that first recognizes the famous ghost shape before reading and decoding the QR code information. Initially the barcodes were a quick method to add friends to your lists (each user has a unique barcode), but the patent drawings also imagined a commercial use case.
Below is a drawing with an advertising use case:
8. Geo-location based ‘pictographs’ aka. Geofilters
In late 2015, Snap was granted its first of several applications related to Geofilter technologies. Based on your location, Snap messages will have the option of filters that display related pictograms (often a fun sign of a user’s city location) superimposed on the message.
Other schematics in an application (#20160085773) show an emoji selection that changes based on the location of the user (they called them geomojis). Snap also went on to buy Bitstrips, the startup behind popular avatar maker Bitmoji, in March 2016 and has since incorporated the technology into messaging.
9. Personalized UI
Getting its start as a successful mobile app, Snap pursued lots of patents related to its widely-discussed UI/UX, which is notoriously difficult for anyone over 25 years old to use. The company’s focus has shifted to making the UX more personalized according to user data, location, favorite friends, and the like.
This patent below was granted August 2016 (#9430783), which employs geo-location to prioritize messages based on your user data (in this case, seeing where you are and suggesting a related event gallery to add a Story):
If you aren’t already a client, sign up for a free trial to learn more about our platform.