How Paradime.io Speeds Up Data Pipeline for Customers Up to 50% without Increasing Costs Using AWS Graviton

In the digital age, teams are more agile and more collaborative than ever. Data analysts are essential to every business. They are a vital bridge, transforming huge amounts of raw business data into relevant and usable data dashboards—enabling business functions such as marketing, sales, product, and customer services to ensure their company runs smoothly, is profitable, and maintains growth.

In 2018, Paradime’s founders, Kaustav Mitra and Fabio Di Leta, worked at at an investment company. Kaustav was tasked with re-architecting the data platform from scratch as the company’s previous solution couldn’t produce the results they needed. While dealing with a host of different tools needed to re-architect the platform, he realized the data analytics team were spending far too much time stitching various applications together. As a result, they ended up with no bandwidth to do mission critical and impactful work.

Kaustav realized, “Because of fragmentation in the data analytics tool chain, people are scrambling all the time just to get their day-to-day work done. How can we bring people and data together so they can get their work done way faster? How can we have a seamless working platform for data analytics folks?”

Kaustav was already familiar with AWS and was greatly inspired by how it is built on the knowledge that seamless integration and easy collaboration are vital to getting things done—faster, more efficiently, and with less stress.

Kaustav explains, “Paradime is our game-changing operating system for data analysts, which like AWS, brings lots of tools together in a single place. This similar mindset is a key reason why AWS was Paradime’s cloud platform of choice, right from day one—especially when matched with the 4x performance and cost savings of AWS Graviton processors.”

A concept born from a real need

From this urgent need, Kaustav & Fabio spent 2020 conceptualizing their idea. Then in 2021 during the pandemic, a pre-seeded Paradime was launched with a team of four—and the very first line of code was written on AWS.

Paradime’s philosophy was also aligned to AWS. To ensure they put the needs of data analysts first, they adopted a “working backwards from customers” approach—mirroring the AWS “working backwards” mechanism that starts not with the technology, but with the customer need, ensuring the customer is always at the heart of any new product or service. Kaustav explains, “The tooling needs to adapt to the needs and day-to-day work of data analysts, rather than the data analysts having to adapt their work to the tooling."

From the beginning, Paradime strove to be the fastest moving innovator in the data analytics market, and that means listening and taking action. "If you listen to your customers day-in, day-out, you get to wow them much faster, and increase the maturity of the platform in step-jumps. We help data analyst teams get their work done faster than any other software out there," Kaustav says.

The problem created by the rapid growth of business data

Paradime describes itself as the AI-powered dbt™ platform for analytics teams. Data Build Tool, or dbt, is a solution to the rapid growth of data within businesses today.

Since the 1970's, SQL code has been the programming language of data analytics, used to ask the database questions so it can give analysts the answers they need. However, with the rapid growth in the amount of data needed to operate competitively—writing, change management, and testing of SQL has become far more complex and time-consuming. Since 2016, dbt has been the tool of choice to help solve this problem, allowing scalable, and auditable SQL code that is easy to build, run, test, and manage over time as the business evolves. However, as the Paradime team knows, dbt is just one part of a wide range of data analyst tasks.

The bridge between data stores and business insights

Data analysts work across many teams, from software-literate data engineers, to non-software-literate business functions such as marketing, sales, product, and customer service. As a result, they need to use systems across the entire end-to-end process, of which dbt is a vital, but single part. Kaustav says, “What they need is a workspace that handles their entire end-to-end responsibilities all in one place—Paradime is that workspace, and that’s what elevates it from other competitors who specialize mainly in dbt.”

There’s so much more that data analysts need to cover. For instance, Paradime helps data leaders understand how fast or slow their data pipelines are working, how much money their team is spending, and how much time they're consuming doing their day-to-day work. Kaustav explains, “Because Paradime has this within easy reach, all in one place, data analytics teams can be 50 to 83 percent more productive than using a fragmented system patched together with off-the-shelf tools, or built internally.”

AWS is the power driving Paradime’s performance

For data analysts, speed matters. Organizations rely heavily on data insights—and as part of Paradime’s pledge to make data analysts’ jobs less stressful, Paradime are always looking for ways to speed up their services while keeping costs down. That’s when they turned to AWS Graviton Processors.

Kaustav explains, “Paradime’s Bolt and CI/CD orchestration framework allows data analysts to automatically run dbt commands on a smooth schedule. We wanted to make it faster, while keeping it at the same price-point for our customers. I was looking at the AWS EC2 family of processors and discovered AWS Graviton offered huge speed improvements for the same price-point as our current x86 processors.”

Maximilian Mitchell, Software Engineer at Paradime, continues, “It was hard not to move to AWS Graviton after we'd done our research, because there's so many perks. In some cases, we've sped up our customers' data pipelines on Bolt by 30 to 50 percent, with over 99.999 percent uptime. This is essential to our customers in vital services like healthcare and banking. Very importantly, we can run jobs at very high frequency to make sure data pipelines are up to date—so much so that they’re effectively real time!”

“We can also help our customers run not only dbt, but other things like data quality and data anomaly tests, which means they can ship trusted data to their stakeholders," Max adds.

AWS Graviton has been a win-win for Paradime and its customers

Paradime has also found that migrating to AWS Graviton has meant big cost savings, allowing them to deliver superior performance to their customers without passing on greater cost. “Before AWS Graviton, we could utilize each node to about 50 or 60 percent. Now, we can utilize to a much higher level, about 85 percent, meaning we only need to spin up around 40 percent less nodes. We now get the same level of performance using less resources, resulting in lower costs,” Max explains.

The advantages go on, “Because we can run our clusters to a much higher utilization, we are able to onboard 40 percent more customers for the same cost as before AWS Graviton. We can now pass that value on to our customers with really competitive pricing compared to our competitors,” Kaustav adds.

“This means we don’t have to force customers into enterprise tiers, allowing us to increase customer acquisition and adoption. We can also deliver higher value without higher prices, meaning our investors are happy,” he elaborates.

Migrating to AWS Graviton the AWS way

Paradime had been with AWS since day one, but the quality of their experience still surprises them. “From day zero, we always had an account manager, even though we were the smallest fish in the pond. We always got equal attention irrespective of the size of our company. For instance, for our migration to Graviton they were so proactive, making sure we had an invite to their AWS Graviton breakfast events,” Max explains.

He continues, “I'm not a really a DevOps engineer, I'm more of a back-end engineer—but I got the opportunity to hang out with experts who give me brilliant advice on how to deploy AWS Graviton, for instance to not completely switch off the x86 builds, and instead build for the two architectures, so we could be more flexible when releasing. This in turn helped us to get the first ARM workloads out within a week, and without causing downtime.”

“Because our applications were originally developed locally on ARM Apple MacBooks (M-series), this allowed us to compile them for AWS Graviton’s ARM architecture, as they were already compatible. This made it easy for us to migrate to Graviton. For a one-month period we ran a crossover between the old and new systems, meaning there was zero downtime to our customers, and we didn't need to do any code changes on our end,” Max adds.

4x performance—enabling more productivity, greater agility, and faster decisions

4x performance with AWS Graviton is an impressive statistic, but how does it benefit Paradime’s customers beyond pure speed? Kaustav explains, “This directly translates into three things; number one is productivity, something that would previously take you four minutes to do, now takes one minute. Number two is agility, teams can do more valuable work with less resources, and so are able to respond to business changes faster. Number three is decision making, teams are able to provide business insights to their end users in a timelier fashion, so they can act more quickly.”

Kaustav continues, “As an example, one of our clients used to get 60 to 80 spreadsheets from various different business units, and each spreadsheet would take 20 minutes to ingest. Along with reporting consolidated financial performance at the end of the month, this entire process would take about a week. With Paradime, it all takes about 20 minutes.”

What’s next for Paradime?

“Our vision for Paradime is to become globally known as the operating system for analytics,” Kaustav exclaims. “So, when people think of data analytics, they think of Paradime, because it takes zero effort to set up and it just works.”

Working towards this, Paradime is focused on improving all aspects of its service. For instance, after migrating to AWS Graviton they have improved their deployment rate by 40 to 50 percent, so they can now release updates more frequently. This means they can add features and respond to customer requests up to two to three times a week—many times quicker than before.

“We know AWS is constantly releasing newer services and features, and we’re always looking to piggyback on new AWS services to provide a really effortless platform to our customers. We have a stable architecture right now, but with AWS we continue to hunt for that extra performance, and they are supporting us in engaging and selling to different enterprises, and scaling our process so we can work with massive enterprises across the globe,” Kaustav explains.

AWS actively supports companies looking to migrate to AWS Graviton, with tools like the AWS Compute Optimizer, and Graviton Fast Start—a program to help move workloads to AWS Graviton. Learn how to start your AWS Graviton migration journey. Find out how.