Jan 17, 2023

January 17, 2023 Building and managing the infrastructure yourself gives you more control – but trying to keep it all under control can take resources away from innovation in other areas. Matt Duka, CTO of FiveStars, a marketing platform for small businesses, doesn’t like that trade-off and goes out of his way to outsource whatever he can. This shows in his reluctance to run his own servers, but it is perhaps most obvious in his approach to data engineering, where he focused on automating or outsourcing mundane maintenance work and data analysis at the end of a five-year journey. Closer to focusing on internal resources. , FiveStars offers small businesses an online loyalty card service – the digital equivalent of a “buy nine, get one free” stamp card – that they can link to their customers’ telephone numbers and payment cards. More than 10,000 small businesses use its services, and Doka estimates that approximately 70 million Americans manage loyalty programs. More recently, it has moved into payment processing, an option adopted by about 20% of its customers, and offers its own PCI-compliant payment terminals. Recording all those interactions generates an enormous amount of data, but that’s not the half of it. To one-up legacy payment processors that leave a terminal unattended and customers to call support if it stops working, FiveStar builds telemetry systems into its terminals, which allow regular Visually reports their connection status, battery level and application performance information. “The bulk of our load isn’t transactions, points or even credit cards,” he says. “There’s a huge amount of device telemetry data to make sure it’s a best-in-class experience when someone wants to make a payment or earn some points.” To figure that out from the data took a lot of analysis—tasks for which the 10-person data team had little time because just maintaining their data infrastructure was eating it all up. The data team that built the first version of FiveStar’s data infrastructure started on the sales and marketing side of the business, not IT. That historical mishap, says Doka, meant that while they really knew their way around data, they had little infrastructure management experience. When Doka took over the team, he found they had written everything by hand: server automation code, database queries, analysis — everything. “He wrote bash scripts!” Doka says. “Even 10 years ago, you had systems that could abstract out bash scripts.” The system was brittle, highly manual and heavily based on tribal knowledge. The net effect was that data analysts spent most of their time just keeping the system running. “They struggle to get new data insights developed in the analysis,” he says. Back in 2019, he says, everyone’s answer to this kind of problem was to use Apache Airflow, an open-source platform for managing data engineering workflows, written and controlled in Python. It was originally developed at AirBnB so the team at Doka was still working hand in hand. Doka opted for a hosted version of Airflow to replace FiveStar’s resource-intensive homebrew system. “I wanted to get us out of the business of hosting our own infrastructure because these are data analysts or even data engineers, not experienced SREs,” he says. “It’s not even a good use of our time.” The adoption of Airflow means that Doka can stop worrying about things other than servers. “There was a huge improvement in the standardization and the basics of running things,” he says. “You’ve inherited all these best practices that we were inventing or re-inventing ourselves.” But, he laments, “how you actually work in Airflow is entirely up to the development team, so you still spend a lot of brain cycles on structuring each new project.” And one of his particular complaints was that you have to create your own documentation of best practices. So barely a year after starting the migration to Airflow, Doka found itself looking for something better to help it automate more of its data engineering processes and standardize some of the less business-critical decisions, including it takes time. They cast their net wide, but the many tools they found only solved part of the problem. “DBT only focused on how to transform data within a snowflake instance, for example,” he says. “It works really cool, but how do you get the data from all your sources into Snowflake?” For that, he says, “there were some platforms that could separate all the data movement in a standardized way, like Fivetran, but they didn’t really give you a language to process.” After checking out several other options, Doka eventually settled on Ascend.io. “I loved the fact that there was a standard way to write an SQL query or Python code, and it generated a lineage and a topology,” he says. “The system can automatically know where all the data came from; How did it make its way to this final analysis? This takes away the challenge of not only running the servers, but also deciding how you operate, he says. “It saves a ton of mental load for data engineers and data analysts,” he says. “They are able to fully concentrate on the question they are trying to answer and the analysis they are trying to do.” He said that not only is it easier for analysts to focus on their work, but it is also easier for them to follow each other. “It’s all documentation that was just created by design, where without thinking about it, each analyst left a clear trail of pieces of how they got to where they are,” he says. “So if new people join the project, it’s easier to see what’s going on.” Ascend uses another Apache project, Spark, as its analytics engine, and has its own Python API, PySpark. It took less than a month to migrate some of the core use cases before Airflow. “Getting Postgres and some of our data sources up and running took an hour and a half,” says Doka. “It was too fast.” Replicating some workflows was as easy as copying the underlying SQL from Airflow to Ascend. “Once we had it working on parity, we would turn it on [old] flow and pour [new] output connector where it needs to go,” he says. The most useful thing about Ascendant was that it would run code changes so quickly that the team could develop and fix things in real time. “The system can be aware of where in the workflow pieces have changed or not, and it doesn’t redo everything if nothing has changed, so you’re not wasting computation,” he says. Huh. “It was a really good pace.” Although some things still involve an overnight wait. “There’s an upstream service that you can only download between 2 a.m. and 5 a.m., so getting that code right, to make sure it’s downloading at the right time of day, was a pain , but it wasn’t necessarily Aarohi’s fault,” he says. Mobilizing a Culture Change The Ascension move did not require any major retraining or recruitment. “Building is pretty much zero now, because we have everything abstracted,” says Doka, “and now there are three people working on top of the new system, and about six analysts reporting and generating insights from the data.” Huh. “Most of the infrastructure work is over,” he says. “There’s still some ETL work, changes and cleanup that never go away, but it’s now done in a standardized way. However, a The thing that took time to digest was that what I call vanilla Python used to be Spark Python with Airflow. It just feels different from writing procedural code.” This isn’t esoteric knowledge, just something the FiveStar team hadn’t used before and needed to familiarize themselves with. A recurring theme in Doka’s data engineering journey is looking for new things that he can stop making and buy instead. “When you build, own and operate the infrastructure in-house, you have a greater level of control and knowledge,” he says. “But often you sacrifice a lot of time for it, and in many cases not having the best expertise to develop it.” Convincing my colleagues about the benefits of downsizing was not easy. “I fought with the team in both eras,” he says. “It’s always part of the transition to any more abstract system.” Doka says he has worked with a number of startups, either as an investor or an advisor, and has always sought the best-in-class technology-minded founders to avoid running the infrastructure and hosting things for them. Class sellers ask to choose – and not just because it saves time. “You’ll also learn better ways to work with them,” he says. He gives the same advice to enterprise IT leaders when working with internal teams. “The most consistent thing I’ve seen in 11 years as a CTO is that gravity pulls people to ‘make it here’ for some reason,” he says. “I never understood it.” This is something that must be constantly resisted or wasted time that is not part of the core business.