Apple’s New Patent Brings 3D Printing To Augmented Reality (AR)

While Apple has had plenty of success in consumer hardware, it has stayed away from 3D printing. Aside from one patent application in 2014 focused on printing multicolored 3D objects, and another for a rapid prototyping technique using liquid metals, Apple hasn’t made any noteworthy plays in the space.

But a newly secured patent could be used to make 3D printing a piece of Apple’s augmented reality initiative: The tech giant recently acquired a patent (US 9,776,364) for an AR-compatible 3D printing system.

The patented technology plugs a gap in current 3D printing techniques: Existing methods have provided no solution for new objects or materials to be 3D-printed onto existing or unfinished objects. For example, printing a new handle onto an existing cup.

The new Apple patent uses visuals – either from cameras or augmented reality systems – to understand the pose and alignment necessary to 3D-print materials onto an existing item, creating what the patent calls called a “composed object.”

The technology could even provide a way for programmers to integrate ARKit-developed apps with 3D printing technology.

Apple’s newly acquired patent comes via metaio, the augmented reality startup Apple acquired in 2015.

Apple’s metaio acquisition was widely seen as laying the groundwork for ARKit, the development framework enabling programmers to build augmented reality apps for iOS.

metaio was also the original applicant on the AR patent, surfaced in July 2017, that hinted at how Apple could use “semi-transparent spectacle or glasses” to survey points of interest in a view of a real environment, via the screen of an iPhone or iPad.

The new patent Apple secured via metaio this October builds on this earlier intellectual property, giving Apple a way to use its augmented reality ecosystem – perhaps even its AR glasses of the future – to support 3D printing.

As the patent explains, existing technologies are insufficient for using 3D printing – aka “additive manufacturing” – to add new layers or objects onto existing ones to create “composed objects” (as with the aforementioned handle-equipped cup).

The newly acquired patent solves that problem by instructing 3D printers and their “printing coordinate systems.” The methods claimed in the patent would enable a 3D printer to recognize what pose and alignment (between existing and to-be-printed objects) are necessary to create a composed object, like the one shown below.

Visuals are required in order for the 3D printing system to recognize the pose and alignment between objects. In the patent’s language, in order to print a “first object” onto an existing “second object,” the system would need “at least one image representing at least one part” of the second object.

Based on the image(s), the 3D printer could compute the appropriate positioning required to print new layers onto the right surface area of the existing object.

To obtain those images, the system utilizes cameras or “any device providing images of real objects.” The schema below explains how the camera (or other device), 3D printer, and “printing coordinate system” would communicate to execute the task of printing new objects onto existing ones.

Ostensibly, the more information a camera or device was capable of providing to the printer and coordinate system, the better: The patent notes that a camera or other device could “further provide an image with depth data,” for example.

That’s fairly key information for 3D printing – especially when creating “composed objects” from new and existing items or materials. Consider our example of a 3D printer adding handles to a cup; to make that possible, the printer must understand depth and dimensions (related to things like cup length, height, and diameter) as well as integrate the cup’s hollowness into the handle design, to make sure the handle isn’t too heavy for the object.

Just as a camera’s depth data aids in that process, augmented reality could prove highly useful for 3D printing composed objects: AR technology provides spatial representations of how objects can sit together in a physical environment, and offers 360-degree viewing to visualize things like depth, dimensions, or hollowness.

The patent recognizes those advantages, noting the use and value of AR for “overlaying computer-generated virtual information” onto real-world views of existing objects. The new composed object might even be viewed in AR through “semi-transparent glasses.”

The mention of glasses is brief, but is a hint that this technology could be integrated to work with the same AR glasses illustrated in the metaio patent that Apple acquired back in July.

The mention of a “mobile device equipped with a camera” (with or without semi-transparent glasses) is also telling, because it hints at potential 3D-printing related opportunities for the ARKit: With the patented innovation, programmers might be able to develop AR apps that leverage the phone’s camera to visualize modifications to existing materials, structures, or items, and integrate their apps to Apple’s 3D-printing systems.

That could have huge value for 3D printing apps that serve a broad variety of use cases – including in industrial manufacturing (for adding elements to existing devices on the assembly line) or medical devices (for 3D-printing specialized modifications onto existing implants, for example).