Amazon Aurora zero-ETL integration with Amazon Redshift enables near real-time analytics and machine learning (ML) using Amazon Redshift on petabytes of transactional data. Within seconds of transactional data being written into Amazon Aurora, zero-ETL seamlessly makes the data available in Amazon Redshift, removing the need to build and maintain complex data pipelines that perform extract, transform, and load (ETL) operations.
Benefits
Near real-time data access
Access transactional data from Aurora in Amazon Redshift within seconds to run near real-time analytics and ML on petabytes of data.
Easy to use
Quickly analyze your transactional data in near real time without having to build and manage ETL pipelines to move transactional data to analytics systems.
Seamless data integration
Consolidate multiple tables from various Aurora database clusters and replicate your data to one Amazon Redshift data warehouse to run unified analytics across multiple applications and data sources.
No infrastructure management
Run near real-time analytics on transactional data without having to manage any infrastructure when using both Amazon Aurora Serverless v2 and Amazon Redshift Serverless.
Use cases
Near real-time operational analytics
Use Amazon Redshift analytics and ML capabilities to derive insights in near real time from transactional and other data to effectively respond to critical, time-sensitive events. Near real-time analytics can help you get more accurate and timely insights for use cases such as content targeting, optimized gaming experience, data quality monitoring, fraud detection, and customer behavior analysis.
Analytics at scale
With the Aurora zero-ETL integration with Amazon Redshift, you can use Amazon Redshift capabilities to analyze petabytes of your transactional data consolidated from multiple Aurora database clusters. You can take advantage of the comprehensive analytical capabilities of Amazon Redshift, such as built-in ML, materialized views, data sharing, and federated access to multiple data stores and data lakes. With Amazon Redshift ML, you can run billions of predictions with straightforward SQL commands with native integration into Amazon SageMaker.
Reduce operational burden
Moving data from a transactional database into a central data warehouse often requires building, maintaining, and operating a complex data pipeline ETL solution. With a zero-ETL integration, you can seamlessly replicate the schema, existing data, and data changes from your Aurora database to a new or existing Amazon Redshift cluster. Zero-ETL integration removes the need for complex data pipeline management.
How to get started
To create your zero-ETL integration between Aurora and Amazon Redshift, you specify an Aurora DB cluster as the data source and an Amazon Redshift data warehouse as the target. The integration replicates data from the source database into the target data warehouse. The data becomes available in Amazon Redshift within seconds, allowing data analysts to begin using Amazon Redshift analytics and ML functionality on the data. To learn more, please visit the getting started guides for Aurora and Amazon Redshift.
Resources
Pricing
- Additional I/O and storage used by enabling change data capture
- Snapshot export costs for the initial data export to seed your Amazon Redshift databases
- Additional Amazon Redshift storage for storing replicated data
- Additional Amazon Redshift compute for processing data replication
- Cross-AZ data transfer costs for moving data from source to target.
Ongoing processing of data changes by zero-ETL integration is offered at no additional charge. For more information, please visit the Aurora pricing page.
Customers
Freight verification is crucial for maintaining supply chain integrity for Infosys' manufacturing, retail, and logistics customers. To achieve this, real-time ingestion of freight tracking data is necessary for tracking shipments and freight analytics.
"Infosys has utilized the capabilities of zero-ETL to ingest data in real-time, enabling freight shipment tracking and providing visibility into location, status, and estimated arrival time. This allows for optimization of freight operations and significant cost reduction. With zero-ETL comprehensive reporting and analytics, along with Infosys Cobalt and Topaz industry blueprints, data-driven decision-making is now possible."
Sunil Senan, SVP and Global Head of Data, Analytics and AI - Infosys
Intuit is the global financial technology platform that powers prosperity for 100 million consumer and small business customers with TurboTax, Credit Karma, QuickBooks, and Mailchimp. With the complexity of an upcoming migration, Intuit was looking at a staggering rate of more than 10 million profile migrations per day. Intuit turned to Amazon Aurora MySQL zero-ETL integration with Amazon Redshift to streamline their data ingestion process and eliminate the need for complex engineering work.
"The sheer volume of data and the need for seamless migration presented a significant challenge for us. Moreover, we needed a solution that would allow us to analyze the migration process rapidly, enabling us to make quick, data-driven decisions. To address these challenges, we turned to Amazon Aurora zero-ETL integration with Amazon Redshift to streamline our data ingestion process and eliminate the need for complex engineering work. With zero-ETL, we were able to send massive amounts of data to Amazon Redshift without the need for data capture or separate ingestion jobs. This allowed for quick insights to drive critical technical and business decisions, saving months of effort that would otherwise have been required. Utilizing Aurora zero-ETL integration with Amazon Redshift helped us explore new patterns at Intuit for large-scale data migrations and near real-time analytics."
Aruna Ghiware, Product Development Manager - Intuit
KINTO Technologies Corporation is a leading player of the mobility platform industry and is the technology company responsible for the development of the KINTO service as Toyota's financial services company. Leveraging the Amazon Aurora MySQL zero-ETL integration with Amazon Redshift, KINTO Technologies was able to achieve a more resilient data pipeline and can now apply Amazon Redshift's advanced analytics features to its operational data in near real-time.
“Prior to the zero-ETL integration being available, we used a custom built solution that continuously streamed changes from our core databases to downstream applications, but we faced persistent performance challenges and impacts to our production workload. To tackle the performance impact on the production workload, we had to manually tune pipelines to send updates less frequently and settle for old data in Amazon Redshift. Using the Aurora MySQL zero-ETL integration with Amazon Redshift, we are able to always have near real-time data available in Amazon Redshift, eliminating developer hours spent manually managing data pipelines for ETL operations or dealing with performance impacts to our workloads, which helps reduce our operational burden.“
Hitoshi Kageyama, Executive Vice President - KINTO Technologies Corporation
Money Forward i strives to boost the productivity and creative potential of IT teams in organizations by streamlining repetitive tasks, cutting costs, and fortifying security through Admina, their intuitive SaaS Management Platform.
“Before the release of Amazon Aurora zero-ETL integration with Amazon Redshift, the burden of implementing and maintaining our ETL operations to analyze product data from Amazon Aurora MySQL in Amazon Redshift was challenging. The Aurora MySQL zero-ETL integration with Amazon Redshift enables near real-time data synchronization between our Aurora MySQL databases and Amazon Redshift, reducing the time to build an analysis environment from a month to just three hours. In addition to reducing the initial burden at time of development, with the zero-ETL integration there is less impact on the production environments, making it possible to build the analysis environment at minimum cost and maximum speed.”
Katsutoshi Murakami, Director and CPO - Money Forward i
Woolworths is a leading sub-Saharan African retailer offering a wide range of quality clothing, general merchandise, and food products with a focus on innovation, value, and sustainability.
"Deriving timely insights from our data is critical, as it means we can quickly promote data-driven decisions across our business and respond effectively to critical, time-sensitive events. Using change data capture and the coagulation of records was time consuming and more costly than expected to process. By using Amazon Aurora MySQL zero-ETL integration with Amazon Redshift, we could produce the same results within a day that otherwise would have taken 2 months to develop due to engineering effort required for ETL operations. Our data latency was significantly reduced using the integration, as the data was in a ready state to query. This led us to make decisions quicker as events were occurring. We lowered our engineering effort, reduced points of failure in pipeline management, and saved costs using the Aurora zero-ETL integration with Amazon Redshift."
Raamy Schroeder, Business Intelligence Lead - Woolworths
Learn more about the features of Aurora