Materialize

Oct 4, 2022

Share Materialize announced early availability of its distributed streaming database, which enables immediate, widespread adoption of real-time data for applications, business functions, and other data products. In an industry first for streaming data, Materialize delivers in a single platform the separation of storage and compute, strict-serializability, active replication, horizontal scalability and workload isolation — all through a simple SQL interface available as a fully-managed cloud service. Materialize is now the fastest way to build products with streaming data, drastically reducing the time, expertise, cost and maintenance traditionally associated with implementation of real-time features. Until now, adopting real-time data has come with sky-high development costs and extreme complexity in implementation, yet still lacks the capabilities necessary to truly productionalize the resulting data products. However, working with streaming data is no longer a nice-to-have for high-performing companies, and in the next decade, companies will have to work with a real-time first approach to their data. Materialize, for the first time, gives users all the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. “Materialize is one of the highest-leverage solutions available in the streaming space,” said Jared Noynaert , Vice President, Data and Analytics at Crane Worldwide. “With the added persistence, high availability, decoupled storage and compute, and improved ergonomics, Materialize delivers the right abstraction at the right time.” Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. It allows developers and data teams to build customer-facing workflows, data engineers to build data applications, and analytics engineers to perform streaming analytics, leveraging integrations with powerful platforms like dbt. Materialize gives developers results that are always up-to-date, enabling them to quickly build automated, low-latency applications downstream. New innovations announced today include: Availability as a fully-managed cloud-native software-as-a-service platform Elastic storage (AWS S3), separated from compute increases scalability and availability while reducing costs Strict-serializability eliminates stale data and enables strong consistency guarantees Multi-way complex joins supports stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL Horizontal scalability leverages Timely Dataflow to let users handle large, fast-scaling workloads Active replication enables users to spin up multiple clusters with the same workload for high-availability Workload isolation enables users to spin up multiple clusters with different workloads while still leveraging shared elastic-storage, enabling collaboration without worrying about interference Related Posts “By abstracting away the tedious stream processing work and allowing both data and software engineers to focus on logic in SQL, we help them create customizable, powerful data experiences, quickly, easily, and cost-effectively,” said Materialize Co-founder and Chief Scientist Frank McSherry. “Real-time products haven’t been impossible to implement, they’ve just been extremely difficult, due to the need for custom development and ongoing maintenance. Standard SQL significantly lowers the bar to engagement and should be sufficient for all but the most complex use cases, enabling valuable engineering resources to be applied to the most sophisticated challenges.” Using standard ANSI SQL and looking and acting like a Postgres database, Materialize, which is built atop Timely Dataflow and Differential Dataflow: Incrementally maintains the results of SQL queries as materialized views, in-memory or on cloud storage, providing millisecond-level latency on complex transformations, joins, or aggregations. Ingests data from multiple sources, including relational databases, event streams, and data lakes before transforming or joining data using the same complex SQL queries used with batch data warehouses. Builds materialized views and incrementally updates the results of as source data changes, rather than computing the answer to a query from scratch every time like a traditional database. Users may either query the results for fast, high-concurrency reads, or subscribe to changes for pure event-driven architectures. “We believe in a future where the default for developers and data teams will be working with data in real-time,” said Arjun Narayan, CEO of Materialize. “The availability of this streaming database platform will help accelerate a migration from batch to real-time in the same way that we saw an enormous shift from on-premises infrastructure to the cloud. Materialize enables companies to not just be data-driven, but to be event-driven, and we provide a critical building block in the creation of these event-driven businesses. If you need to build something real-time, you should build with Materialize first.”