Bringing the HBO comedy Silicon Valley's "Shazam for Food" to life, Microsoft was granted a patent for augmented reality glasses that track when and what you eat.

Imagine a little voice in your ear telling you that actually you shouldn’t get that side of fries. Today, Microsoft was granted a patent for a wearable system for imposing dietary self-control. These are augmented reality glasses that can monitor what the wearer eats including calorie content and makes food recommendations based on nutritional needs as well as product and restaurant reviews.

Easy, accurate nutritional tracking has been a long-time goal for many companies and consumers, and this patent is just the newest example of tech giants entering the fray. Apple recently patented a system of RFID tags to track the nutrition of restaurant orders, for example. Microsoft previously filed several patents related to diet, including an allergy awareness system, and a platform to analyze restaurant menus based on user preferences. (Links are accessible to CB Insights clients).

Disruption in Digital Health Webinar
Learn how startups and private market trends are changing the $3T healthcare industry.

Microsoft’s new patent, titled “Wearable food nutrition feedback system,” includes mixed-reality glasses that can monitor sight, sound, location, temperature, and motion. These sensors help the glasses identify food in front of the wearer and analyze its nutritional content. Eyeball tracking helps the glasses understand which items the wearer is considering. Based on the wearer’s needs — including any special diets, calorie restrictions, allergies, or personal flavor preferences — an augmented reality display can steer wearers away from or toward certain foods.

This Microsoft patent was filed on 10/8/15 and granted on 5/9/17, and is an extension of an earlier patent filed in 2012.

Microsoft ar diet glasses flowchart pic

Beyond making recommendations in the moment, it can also track users’ caloric and nutritional intake over time. The glasses use gesture recognition to confirm which foods the wearer chooses, and then monitor sound and head movement to track how much the wearer actually eats (the “consumption event,” as the patent puts it). For example, if the glasses track a user’s hand picking up a hamburger, and then see the hamburger returned to the plate with a bite taken out, it will confirm what and how quickly the wearer is eating. On the other hand, this may mean you can’t try to fool it by staring at a plate full of broccoli while reaching behind your back for the pizza.

The ability to easily track caloric intake has been a challenge for current wearable fitness trackers, such as Fitbit. For now, many companies offer solutions such as manual input or barcode scanning. In addition, many wearables on the market focus on counting steps and other exercises — calories out, not calories in.

After identifying the foods using the “image processing engine,” Microsoft’s system matches them to nutritional data provided by the manufacturer, submitted by the restaurant, or scraped from similar recipes. As the image below highlights, the heads-up display can warn users about nutritional limits or make suggestions when the user is looking at the food.

Patent pic 2nd big meal

Apart from nutrition, Microsoft also aims to integrate social data to make personalized recommendations. For example, once GPS data confirms the wearer has entered a certain restaurant, the device could pull in any reviews written by friends. Users could also choose to share more detailed information on their past meals with specific other users, or could pull up their own past information, as shown below.

Menu social historical feedback pic


Want more patent innovation data? Log in to CB Insights or sign up for free below.

  • Thomas Leavitt

    No. Just… no. In every respect… the patent, the concept, the implementation. I have serious doubts that even the most desperate dieter is going to accept this level of intrusive self-monitoring. This reads like something out of a cheesy dystopian sci-fi movie from the 1990s.

  • SheldonS

    If it could accurately calculate the total carbohydrates in the food that would be a great help for Type 1 Diabetics. It can be a challenge to figure out or guess at the total carbohydrates minus dietary fibers in a prepared dish.