Palantir is one of the world's most opaque and influential data companies. We dig into its patents as Facebook's data privacy scandal refocuses media attention on how personal data is used in business and beyond.
Palantir Technologies, a privately owned Silicon Valley-based tech company, sells a suite of software for integrating, visualizing, and analyzing the world’s data.
Little is known about Palantir’s technology or products. Recently, reports surfaced that an employee of Palantir helped Cambridge Analytica mine millions of Facebook profiles in 2014, which could deepen scrutiny of Palantir.
Given the secrecy surrounding the company, any visibility into the company’s methods is useful in catching a glimpse into how it goes about mining data and extracting value from it for clients.
A cluster of recently published patent applications— two of them just last week — show that:
- Palantir wants to be able to parse datasets as large as trillions of email messages.
- It wants to use household data such as income and other signals to predict consumers’ future actions, such as cancelling a subscription service.
- In a very “Minority Report” type patent, Palantir also wants to use data points connected to individuals to predict future criminal activity.
Together, Palantir’s patent applications provide forward-looking insight into the company’s corporate product strategy and innovation, as well as where bleeding-edge population-data and analytics technology is headed. While an individual patent might only be a sketch of a potential product, taken together, activity across several patents can reveal strategic direction and priorities.
It’s worth noting that Palantir’s products and services are already deployed in the military, intelligence, law enforcement, and financial communities to fight crime.
Indeed, Palantir was partially seed-funded by the CIA’s venture arm, In-Q-Tel. And Palantir just won an $876M contract to provide data management software to the U.S. Army.
Also note that Palantir has been granted patents in the following areas over the last several years. However, the patent applications we investigate below detail the most recent iterations of those designs.
Mining and combining the world’s data:
Patent application published: March 22, 2018
Palantir is building technology to mine and analyze multiple petabytes of data. That’s lots of data. For reference, neurologists estimate that the human brain’s ability to store memories is equivalent to about 2.5 petabytes. Facebook hosts around 300 petabytes of data in total. Palantir can mine both structured and unstructured data from the internet, as well as integrate the databases of third-parties into its system.
“The input data may include e-mail message spools, transaction logs, call data records, etc. that might be found in a business, organization, governmental agency, school, university, hospital, etc.”
The invention outlined in the patent application is capable of “ingesting data from disparate heterogeneous data sources in a single data repository.” The application notes that input data may include transaction logs of banks, call data records, computer network access logs, e-mail messages of a corporation, or other potentially high-volume data that may contain up to billions to trillions of records.
Palantir’s overall goal is to be able to take that data and efficiently identify information of interest. With this tool “analysts can investigate top-down trends, behaviors, and activities, or conduct bottom-up target centric analysis”.
Predicting the future one household at a time:
2. System and method for determining a propensity of an entity to take a specified action
Patent application published: March 22, 2018
Palantir’s prediction tech builds off the mantra that past behavior is the best predictor of future behavior. Specifically, the tech described can acquire data on a subject, create a record of the subject’s behavior, extrapolate past behaviors into “features” that color the subjects’ actions, and then use the data to train a prediction model that spits out the likelihood that the subject will take specific actions in the future.
The example of predicting behavior detailed in the patent application includes ingesting data on average household income, number of cars, and bills to pay. That information can let an analyst see if the household is likely to cancel a subscription to a service within the next month. In the sketch below, the household being profiled is deemed to have a “medium” propensity to churn.
It’s easy to see why a technology of this type would be valuable to corporations with a business-to-consumer subscription business model such as telecommunications and media companies.
Fighting crime before it happens:
Patent granted: December 5, 2017
Palantir has put its predictive policing tech to use. The company was reportedly contracted by the New Orleans Police Department in 2012, to help the department identify targets that are likely drivers of violence. That contract is reportedly not being renewed by the city due to its controversial nature. Similar contracts still exist between Palantir and law enforcement agencies in Los Angeles and New York, though details on the status of those contracts are secret.
Palantir’s intention for its crime risk forecasting tech is to help police “to know when and where crimes are most likely to occur in the future.” The crime prediction algorithms are enabled by advances in machine learning. The algorithms enable police officers to “effectively gauge both the level of increased crime threat and its potential duration.” Police can determine crime forecasts for specific areas, time-frames, and even for specific crimes.
