Migrate from Apache Cassandra to Amazon Keyspaces

If you need to create a source Cassandra database for the migration walkthrough, go to the Amazon EC2 console. Choose Launch instance to start the Amazon EC2 instance creation wizard.

The first step in the Amazon EC2 instance creation wizard is to choose your Amazon Machine Image (AMI). Use the Amazon Linux 2 AMI with the default x86 architecture and choose Select.

On the next page, choose the instance type for your Amazon EC2 instance. For this walkthrough, use at least a t2.large instance because Cassandra requires a significant amount of memory. Choose Review and Launch to continue.

The next page shows the default options for the rest of your Amazon EC2 settings. The default options should be fine for this use case. Choose Launch to continue.

Now choose a key pair to allow SSH access to your new Amazon EC2 instance. From the dropdown, choose Create a new key pair to create a new key pair for this walkthrough. Then name your key pair cassandra-migration and choose Download Key Pair to download the key pair to your machine. Finally, choose Launch Instances to create your instance.

A confirmation page shows that your instance is launching. Choose View Instances to see your Amazon EC2 instance.

As your instance is initialized, it shows an Instance State of pending. Wait until the Instance State shows running.

When the Instance State shows running, you can SSH into your instance.

Copy the IPv4 Public IP value for your instance, and then run the following commands in your terminal to SSH into your instance.

chmod 600 /path/to/cassandra-migration.pem
ssh -i /path/to/cassandra-migration.pem ec2-user@<IPv4PublicIP>

Be sure to use the proper values for the path to the cassandra-migration.pem file that you downloaded and use the correct IPv4 Public IP for your instance.

If you have difficulty connecting to your Amazon EC2 instance, see Connecting to your Linux instance using SSH.

If you connected successfully, your terminal should look like the following.

To install Cassandra, you first need to install Java. To install Java, execute the following commands in your terminal.

sudo amazon-linux-extras enable corretto8
sudo yum -y install java-1.8.0-amazon-corretto-devel

After you have installed Java, execute the following commands to install and start Cassandra.

echo '[cassandra]
name=Apache Cassandra
baseurl=https://www.apache.org/dist/cassandra/redhat/311x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://www.apache.org/dist/cassandra/KEYS' | sudo tee -a /etc/yum.repos.d/cassandra.repo > /dev/null
sudo yum -y install cassandra
sudo systemctl daemon-reload
sudo service cassandra start

After you have installed and started Cassandra, use the nodetool CLI to check the health of your Cassandra cluster. In your terminal, execute the following command.

nodetool status

You should see a single node in your Cassandra cluster like the following.

There is an open-source tool called tlp-stress that is used for load testing and benchmarking your Cassandra cluster. It includes a simple way to load your cluster with data that you can use to ensure that your migration works.

To install tlp-stress, run the following commands in your terminal.

sudo yum install -y git
git clone https://github.com/thelastpickle/tlp-stress.git
cd tlp-stress
./gradlew shadowJar

Next, use tlp-stress to load your cluster with 10,000 time-series records for the migration. To do so, execute the following command in your terminal.

bin/tlp-stress run BasicTimeSeries -i 10k

You should see some output in your terminal as the command is executed. When the command has run, it will show the results of your operations.

To get started, navigate to the Amazon Keyspaces console. On the Keyspaces page, choose Create keyspace to create a new keyspace.

In the keyspace creation wizard, give your keyspace a name.

You can attach tags to your keyspace to help with access control or to track billing. Then choose Create keyspace to create your keyspace.

At this point, you should see your keyspace in the Amazon Keyspaces console. Choose your keyspace to view more details about it.

Currently, your keyspace does not have any tables. Let's create the first table in your keyspace that will hold your migrated data.

Choose Create table to open the table creation wizard.

Choose a name for your table. The tlp-stress tool gave your table a name of sensor_data, so use the same name here.

Now you need to declare the schema for your table. The tlp-stress tool added three columns to the table: sensor_id, timestamp, and data. The sensor_id and data columns are of type text, and the timestamp column is of type timeuuid. Additionally, the partition key is a combination of sensor_id and timestamp.

Your schema should look as follows in the Amazon Keyspaces table creation wizard.

Finally, you can choose the Capacity mode and add any required tags. With Amazon Keyspaces provisioned capacity billing mode, you declare the amount of reads and writes you want to provision. With on-demand billing mode, you don't need to plan for the capacity required by your table. Amazon Keyspaces bills you directly for the reads and writes you consume. As part of the AWS Free Tier, you can use 30 million on-demand read units and 30 million on-demand write units per month for the first three months after you create an Amazon Keyspaces resource.

Notice that when creating your Amazon Keyspaces table, you did not need to configure a compaction strategy, bloom filters, caches, or other common tuning parameters as you might in Cassandra. This is handled for you by Amazon Keyspaces so that your developers can focus on the quality of the data model and your required access patterns.

The table creation wizard shows you the Cassandra command that will be executed to create your table. When you are ready, choose Create table to create your table.

The Amazon Keyspaces console shows your table being created. While it is being created, your table has a Status of Creating.

When your table is ready to use, its Status is Active.

Use cqlsh, the command-line tool for working with Cassandra, to assist with the migration. First, export the data from your existing table in Cassandra. Then, load the data into your new table in Amazon Keyspaces.

Before you begin, generate service-specific credentials to connect to Amazon Keyspaces by using cqlsh. These service-specific credentials are one of the two ways you can authenticate to your Amazon Keyspaces table. Service-specific credentials are credentials tied to a specific AWS Identity and Access Management (IAM) user that are used to authenticate for a service.

To generate service-specific credentials, navigate to the IAM console. Find the IAM user to whom you want to grant service-specific credentials and choose that user.

On the IAM user's page, choose the Security credentials tab.

Then navigate to the bottom of the page. In the section for Amazon Keyspaces, choose Generate credentials to create Amazon Keyspaces credentials for your IAM user.

A window is displayed with your service-specific credentials. Download these credentials and make sure you have them available because you will need them later in this module.

After you have downloaded your service-specific credentials, you are ready to start the migration. You will perform the migration from the Amazon EC2 instance that is hosting your self-managed Cassandra database.

Navigate to the Amazon EC2 console, and find the Amazon EC2 instance you created in the first module in this lesson.

Copy the IPv4 Public IP address of your instance.

When you have the IPv4 Public IP address, run the same command you ran in the first module to connect to your instance.

ssh -i /path/to/cassandra-migration.pem ec2-user@<IPv4PublicIP>

After you have connected to your instance, enter cqlsh in your terminal to enter the CQL shell.

You can view some of your sample data by using the following command in cqlsh.

SELECT * FROM tlp_stress.sensor_data LIMIT 5;

You should see output similar to the following.

In the cqlsh tool, enter the following command to export your table to a .csv file on your Amazon EC2 instance.

COPY tlp_stress.sensor_data TO 'sensor_data_export.csv' WITH HEADER=true;

It should take a few seconds to complete the command. Exit the CQL shell by typing exit.

There is a file in your current directory called sensor_data_export.csv, which contains the contents of your table. You can view some of the contents by running the following command in your terminal.

head -n5 sensor_data_export.csv

This prints out the header and the first four rows of data.

Now, use cqlsh to connect to your Amazon Keyspaces table by using the service-specific credentials you created.

First, download the Amazon digital certificate with the following command.

curl https://www.amazontrust.com/repository/AmazonRootCA1.pem -o /home/ec2-user/.cassandra/AmazonRootCA1.pem

Then run the following command to configure cqlsh to connect to Amazon Keyspaces.

echo '[connection]
port = 9142
factory = cqlshlib.ssl.ssl_transport_factory

[ssl]
validate = true
certfile = /home/ec2-user/.cassandra/AmazonRootCA1.pem' >> /home/ec2-user/.cassandra/cqlshrc

With cqlsh configured, run the following command to connect to your keyspace by using cqlsh.

cqlsh cassandra.us-east-1.amazonaws.com 9142 -u <user> -p <password> --ssl

Make sure to substitute for <user> the user name and for <password> the password from your service-specific credentials.

If the command was successful, you should be connected to your keyspace by cqlsh.

Navigate to the Amazon EC2 console. Find the instance that is used to host your Cassandra database cluster. Choose the instance, and then choose Instance State > Terminate in the Actions dropdown.

If you no longer need the keyspace and table that you created in this lesson, you should delete those as well. To do so, navigate to the Amazon Keyspaces console. Choose the keyspace you created, and then choose Delete.

A confirmation window is displayed before you delete the keyspace. Type "Delete" in the box, and then choose Delete keyspace to delete your keyspace.

The Amazon Keyspaces page shows that your keyspace is being deleted.