Version: Develop

Migration

This document describes how to migrate data from an older a9s Redis version to an a9s KeyValue. We present two approaches:

the first one uses the Disaster Recovery feature to create a fork of an existing Service Instance,
the second one uses different commands via the Redis/Valkey CLI to migrate the data.

The following migration paths have been tested and are supported by the a9s Data Services Framework:

a9s Redis 7 -> a9s KeyValue 8

Migrate to a9s KeyValue Using the Disaster Recovery Feature

Follow the steps described in Disaster Recovery to create a fork of an existing a9s Redis Service Instance.

Migrate to a9s KeyValue Manually

In the following we present two approaches to migrate to a9s KeyValue manually:

the first one uses the MIGRATE command,
while the other one performs an element-wise DUMP & RESTORE.

Prerequisites

Install redis-cli or valkey-cli.

Known Issues

Downtime during the migration. It will depend on the amount of data inside your a9s Service Instance and network connection throughput between both a9s Service Instances.
Every scenario in this migration was tested with a9s Redis/KeyValue instances following best practices for key naming. So, it is known that there may be problems when trying to migrate that when key names do not follow best practices. An example of this limitation is the use of keys with names containing binary content or other special characters which can potentially cause issues during the migration. Please note that you might need to perform individual migration for any existing special keys, although we do provide an alternative Ruby-based solution in Migrate Using Manual DUMP & RESTORE for such cases.
If a a9s Redis/KeyValue Service Instance uses multiple logical database, in order to migrate the whole Redis/Valkey content, it is necessary to migrate the Redis/Valkey logical database one after another. A piece of information about logical databases:

Out of the box, a Redis/Valkey supports 16 logical databases. These databases are effectively siloed off from one another, and when you run a command in one database, it doesn't affect any of the data stored in other databases in your a9s Redis/KeyValue Service Instance.

Before Migrating

There are some preliminary measures required before the migration can take place:

Isolating The Nodes Before The Migration
Create a New a9s KeyValue Destination Instance
Create an SSH tunnel

Isolating The Nodes Before The Migration

info

It is highly recommended to isolate the a9s Redis/KeyValue Service Instance before starting the migration. This avoids potential data inconsistencies that could be caused if the a9s Redis/KeyValue Service Instance is receiving new data while processing the migration at the same time.

This can be achieved by unbinding all applications from your a9s Redis/KeyValue Service Instance.

Save the RDB File (Redis/Valkey Dump File)

We recommend saving your Service Instance's RDB file as failsafe in the event of unexpected data loss. This file must be stored safely, at least until the consistency of the database has been confirmed, post-migration.

You can download the latest backup from a9s Service Dashboard.

Save the RDB File (Alternative)

Alternatively, it is possible to download the RDB file for a specific Service Instance using `valkey-cli`:

# set `tls` value if the Origin Service Instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
valkey-cli --user <a9s-instance-origin-username> -h <a9s-instance-origin-host> -p 6379 \
  -a <a9s-instance-origin-password> ${origin_tls:-} --rdb /tmp/dump.rdb

Please refer to the Placeholder Values table at the end of this guide for further details on <a9s-instance-origin-host>, <a9s-instance-origin-username>, <a9s-instance-origin-password>, <origin_ca_cert_path> and similar placeholders on this document.

Create a New a9s KeyValue Destination Instance

Create a new and empty a9s KeyValue destination Service Instance that will be the target for the data migration from an existing a9s Redis/KeyValue origin Service Instance.

Note

Do not insert any data inside of the machine before the migration and checking are done.
Create a Service Instance with the same capabilities or greater.

Create an SSH Tunnel

Create a reverse tunnel

The goal is to give access to the Application Developer to the a9s KeyValue Service Instance. And for this, it is necessary to make a SSH tunnels to the a9s Redis/KeyValue origin Service Instance.

In the end, we want to achieve the following scenario below.

                    (port 6379)                             (port 6379)
           /---> * Origin Instance * ----- MIGRATE -----> * Destination Instance *
          /
* CF Application * (via SSH Tunnel)               [infrastructure network]
--------/-----------------------------------------------------------------
       /                                               [Developer network]
* Developer Machine * (local port 6379)

It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the service for any purpose such as debugging. Cloud Foundry (CF) provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.

First of all you must have an application bound to the service. How to do this see Bind an application to a Service Instance.

note

cf ssh support must be enabled in the platform. Ask your platform operator if you are not sure.

Follow the section Create a Tunnel to The Service to create the reverse tunnel.

Migrate to a9s KeyValue Using the MIGRATE Option

Warning Notes

Migrating between an SSL Service Instance and a non-SSL Service Instance, or vice versa, is not supported by a9s Redis/Valkey Service Instance.
When migrating data between SSL Service Instances they MUST have the same CA, or it MUST be possible to verify the certificate of the destination with the CA of the origin Service Instance. Otherwise, it will fail because the TLS connection can not be verified.

For all of the cases above, please use the Migrate Using Manual DUMP & RESTORE.

The main strategy to migrate the whole dataset from one a9s Redis/KeyValue Service Instance to a9s Valkey service instance is based on the Redis MIGRATE feature/Valkey MIGRATE feature.

This feature comes in the form of a simple command that can be run through redis-cli/valkey-cli. Using the CLI, we shall connect to the existing instance, and execute the MIGRATE command. The command accepts the hostname and credentials for the new instance, which implies that there will be a direct connection between the two Service Instances during the migration.

The vendor documentation of the feature also says:

The command is atomic and blocks the two instances for the time required to transfer the key, at any given time the key will appear to exist in a given instance or the other instance unless a timeout error occurs.

Hence, there will be a system downtime during the migration. Thus it is suggested to trigger the migration process during an appropriate time window; likely in the moment in the week, the day, and the period with less impact for the client of this specific a9s Service Instance. The Application Developer must decide which timeframe fits these criteria.

Migration Duration

Based on internal and isolated tests, migrating 1GB takes about 50 seconds on average. However, it will depend on each environment and its particularities. Be aware that it might take some time, so it is necessary to plan wisely.

Perform the Migration

Use the valkey-cli to retrieve the keys and migrate between the a9s Redis and Valkey Service Instances. Replace all placeholder values in the command below and execute it afterwards. These commands may change depending on the version of Redis/Valkey, make sure to run the appropriate command for your case. This command is valid for Redis 7 GA and above:

# set `tls` value if the origin Service Instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

client_origin_command="valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> --user <a9s-instance-origin-username> -a <a9s-instance-origin-password> -n <a9s-instance-origin-logical-database-number> ${origin_tls:-}"

${client_origin_command} --raw KEYS '*' | xargs ${client_origin_command} MIGRATE \
<a9s-instance-destination-host> <a9s-instance-destination-port> "" <a9s-instance-origin-logical-database-number> <timeout> \
COPY AUTH2 <a9s-instance-destination-username> <a9s-instance-destination-password> KEYS > /tmp/migration-output-checker.txt

The migration-output-checker.txt contains the output of the MIGRATE command, and it is used in a further step to make sure the migration was executed with success. For more information see Check The Database Consistency.

The command above can only be executed by an Application Developer once they have access to the a9s origin service instance via the reverse tunnel.

Furthermore, this command is composed of multiple commands, all of which serve a specific function:

The first command, valkey-cli --raw KEYS '*', lists all the keys stored in the existing instance.
The second command, xargs valkey-cli MIGRATE <destination-host> <destination-port> "" 0 5000 COPY AUTH <pass> KEYS, uses the output of the first command to MIGRATE the data of the source host to the destination host.
- The part following the first command and preceding the second one simply utilizes the tunnel to connect to the existing instance, where the second command is executed.
The COPY argument must be present in the command to avoid deleting the key from the source host.

Please refer to the Placeholder Values table at the end of this guide for more specific details on the placeholders.

Logical Databases

If you are using a logical database, and you have more than one, this command must be executed as many times as the amount of the logical databases. It is necessary to change the <a9s-instance-original-logical-database-number> value accordingly.

Key Names

If the origin a9s Redis/KeyValue instance contains keys with names containing special characters this migration process will not work. For example, if one key contains a breakline character this can be a problem.

Check The Database Consistency

Check the database consistency between the original Redis Service Instance and the new Valkey Service Instance databases following the steps below.

Also check the consistency using the Check The Database Consistency (General Approach).

Check the Output of MIGRATE Requests at `migration-output-checker.txt`

The migration-output-checker.txt content is the output result of each MIGRATE request made to the original service instance. It must contain only "OK".

Check the content:

cat /tmp/migration-output-checker.txt | grep --invert-match "OK"

The output of this command must be empty, otherwise it will return an error message with details on what went wrong during the migration. Below we list some of the potential error messages:

Error Message	Possible Cause	Troubleshooting
`NOKEY`	The key does not exist in the origin Service Instance.	Double check if the first part of the command (`valkey-cli --raw KEYS ''`) is correct. The output of this command must have all* keys present in the a9s Redis/Valkey Service Instance.
`IOERR error or timeout connecting to the client`	It is likely a communication problem between the Redis and Valkey instances.	Double check if the origin Redis Service Instance has access to the destination Valkey Service Instance.
`ERR Target instance replied with error: ERR invalid password`	Authentication does not work.	Double check if the password is correct.

Migrate Using Manual DUMP & RESTORE

In case the method using MIGRATE does not work, we can make use of what actually takes place behind the scenes in the MIGRATE command, albeit without assured atomicity and a longer duration:

DUMP - every key in the existing instance is extracted using this command
RESTORE - every key (with data) is reinstated in the new instance

As in the previous examples, for the following scripts you need to make sure to replace the given placeholders (see table below) with the actual values to create tunneled connections to the a9s Redis and a9s KeyValue Instances. We provide two alternatives, the first one being a BASH script while the second one uses Ruby instead. Due to BASH's non-binary-safe nature, we provide the Ruby script, as it can handle the migration of certain binary values or keys that the first script cannot.

Perform The Migration - Non-Binary-Safe Script

The following script (migrate_manual_dump_restore.sh) will handle this task.

Version Warning

Please, note that if an a9s Redis 7 RC version or older is either the origin or destination, the --user <a9s-instance-origin-username> parameter MUST NOT be passed in the commands.

#!/bin/bash

# set `tls` value if the Origin Service Instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
export redis_origin_instance="valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> -a <a9s-instance-origin-password> --user <a9s-instance-origin-username> ${origin_tls:-}"

# set `tls` value if the Origin Service Instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
export valkey_destination_instance="valkey-cli -h <a9s-instance-destination-host> -p <a9s-instance-destination-port> -a <a9s-instance-destination-password> --user <a9s-instance-destination-username> ${destination_tls:-}"

for key in $(${redis_origin_instance} --scan); do
    # For Linux, use `head -c-1` instead of `ghead -c-1` in the command below.
    ${redis_origin_instance} --raw DUMP "${key}" 2> /dev/null | ghead -c-1 > /tmp/operation_dump
    ttl=$(${redis_origin_instance} --raw TTL "${key}" 2> /dev/null)
    case ${ttl} in
        -2)
            echo "Key ${key} not found in origin instance.. Skipping."
            ;;
        -1)
            ${valkey_destination_instance} DEL "${key}" > /dev/null 2>&1
            restore_output=$(cat /tmp/operation_dump | ${valkey_destination_instance} --no-auth-warning -x RESTORE "${key}" 0)
            [ "${restore_output}" != "OK" ] && \
              echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue

            echo "${key} migrated successfully!"
            ;;
        *)
            ${valkey_destination_instance} DEL "${key}" > /dev/null 2>&1
            restore_output=$(cat /tmp/operation_dump | ${valkey_destination_instance} --no-auth-warning -x RESTORE "${key}" "${ttl}")
            [ "${restore_output}" != "OK" ] && \
              echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue

            echo "${key} [TTL = ${ttl}] migrated successfully!"
            ;;
    esac
done

Execute the script:

bash migrate_manual_dump_restore.sh > "/tmp/migrate-manual-dump-restore.txt"

Important Details

The script above uses a third command called TTL. This command reads the Time-to-Live specification of a key in your existing instance. Using this command we make sure to ensure the original expiration time is respected in the new instance.
The script relies on a shell utility called head. If you are using a MacOS environment, you need to install its GNU-based alternative called ghead.
Since this is a manual migration, it is strongly suggested that you isolate the nodes, see Isolating The Nodes Before The Migration for more information. This is necessary, because unlike the MIGRATE command, this script does not ensure atomicity.
Due to the details mentioned above, this script should only be used as a last resort.

Perform The Migration - Binary-Safe Script

The migrate_manual_dump_restore.sh script in the previous section is not able to correctly handle cases of Keys and/or Values having a binary structure. That is because the script will normally store the Keys and Values in variables in a non-binary-safe manner. This becomes an issue when those values are then provided to the valkey-cli in commands like RESTORE or even MIGRATE, which we describe above. To address these issues, we provide a Ruby script below which uses the DUMP & RESTORE idea while being binary-safe. We will use the name migrate_manual_dump_restore.rb.

require 'redis'

origin_instance = Redis.new(host: <a9s-instance-origin-host>, port: <a9s-instance-origin-port>, db: <a9s-instance-origin-logical-database-number>, password: <a9s-instance-origin-password>)
# Example when using SSH tunnel:
# origin_instance = Redis.new(host: 'localhost', port: 6379, db: 0, password: 'a9sa860dc0ef50586d8fe80dcc4db7633672ccccccccc91db1e24fdfd0b59d')

destination_instance = Redis.new(host: <a9s-instance-destination-host>, port: <a9s-instance-destination-port>, db: <a9s-instance-origin-logical-database-number>, password: <a9s-instance-destination-password>)
# Example when using SSH tunnel:
# destination_instance = Redis.new(host: 'localhost', port: 6380, db: 0, password: 'a9sd172addf8c320b20961daaaeb11cfd4d5f3656adc11b75gr6144e07547e5')

origin_instance.scan_each { |key|
  begin

    ttl = origin_instance.ttl(key)

    if ttl == -2 then
      next
    elsif ttl == -1 then
      ttl = 0
    else
      ttl = 1000 * ttl
    end

    dumped_value = source_instance.dump(key)
    destination_instance.restore(key, ttl, dumped_value, :replace => true)

  rescue Redis::CommandError => e
    puts "Migration ERROR restoring key: #{key} - #{e.message}"
  end
}

source_instance.quit
destination_instance.quit

In order to execute the script above, you need to have Ruby 3.x installed in your local environment. You also need to install a gem called redis, by running gem install redis. The gem utility should be present in your environment once you install Ruby. Once you have the local environment setup, you need to create the SSH tunnels as described at the beginning of this guide in Create Tunnel.

Then you need to use the correct Placeholder Values and execute the following command in the command-line:

ruby migrate_manual_dump_restore.rb > /tmp/migrate-manual-dump-restore.txt

Important Details

The script above uses the same logic as the BASH script in Perform The Migration - Non-Binary-Safe Script, so atomicity is again not guaranteed.
Similarly, you are strongly advised to consider Isolating The Nodes Before The Migration before proceeding.
Unless an a9s Redis 7 RC version or older is either the origin or destination, the --user <a9s-instance-origin-username> parameter has to be included in the commands.

Checking For Migration Errors

The migrate-manual-dump-restore.txt contains the output of the execution of any of the scripts above, and it is used in the next step to make sure the migration was executed with success.

Check The Database Consistency

Check the database consistency between the original a9s a9s Redis/KeyValue Service Instance and the a9s Valkey service instance databases following the steps below.

Also check the consistency using the Check The Database Consistency (General Approach).

Check the Script's Output at `migrate-manual-dump-restore.txt`

The migrate-manual-dump-restore.txt content is the output result of migrate_manual_dump_restore.sh script. It MUST not contain errors.

Check the content:

cat /tmp/migrate-manual-dump-restore.txt | grep "Migration ERROR"

Should a problem occur at any point of process, this might indicate that the migration did not finish successfully, in this case make sure you save a copy of the migrate-manual-dump-restore.txt for further investigation and contact your platform operator.

Check The Database Consistency (General Approach)

Check the database consistency between the original a9s Redis/KeyValue Service Instance and the a9s Valkey service instance databases.

Compare Data Between the Service Instances

Limitations

If by any chance the origin Redis Service Instance was not isolated and kept receiving new data during the migration the following check will not work.

Execute the info keyspace valkey-cli command under each host. The keys and expires values must be equal. This command should only be run on a9s Redis 7 GA and above:

# Redis Origin

# set `tls` value if the Origin Service Instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> --user <a9s-instance-origin-username> \
  -a <a9s-instance-origin-password> ${origin_tls:-} info keyspace

# Valkey Destination

# set `tls` value if the Origin Service Instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
valkey-cli -h <a9s-instance-destination-host> -p <a9s-instance-destination-port> --user <a9s-instance-destination-username> \
  -a <a9s-instance-destination-password> ${destination_tls:-} info keyspace

Execute the info memory valkey-cli command under each host. The values must be similar with only a few kilobytes of difference, if any.

# Redis Origin

# set `tls` value if the Origin Service Instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> --user <a9s-instance-origin-username> \
  -a <a9s-instance-origin-password> ${origin_tls:-} info memory | grep "used_memory:\|used_memory_human"

# Valkey Destination

# set `tls` value if the Origin Service Instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
valkey-cli -h <a9s-instance-destination-host> -p <a9s-instance-destination-port> --user <a9s-instance-destination-username> \
  -a <a9s-instance-destination-password> ${destination_tls:-} info memory | grep "used_memory:\|used_memory_human"

Compare Keys Between the Service Instances

This step will compare all keys between both Service Instances.

Get all origin Service Instance keys and sort them.

# set `tls` value if the Origin Service Instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> --user <a9s-instance-origin-username> \
  -a <a9s-instance-origin-password> ${origin_tls:-} --raw keys '*' > /tmp/instance-origin.txt
sort --parallel=2 -uo /tmp/instance-origin-sorted.txt /tmp/instance-origin.txt

Get all destination Service Instance keys and sort them.

# set `tls` value if the Origin Service Instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"

valkey-cli -h <a9s-instance-destination-host> -p <a9s-instance-destination-port> --user <a9s-instance-destination-username> \
  -a <a9s-instance-destination-password> ${destination_tls:-} --raw keys '*' > /tmp/instance-destination.txt
sort --parallel=2 -uo /tmp/instance-destination-sorted.txt /tmp/instance-destination.txt

Compare both Service Instances keys

diff /tmp/instance-origin-sorted.txt /tmp/instance-destination-sorted.txt

The output MUST NOT have differences.

Placeholder Values

Placeholder	Description	Example	Required?
`<a9s-instance-origin-host>`	The domain or ip of the host that contains data to be migrated.	`localhost` or `127.0.0.1`	Yes
`<a9s-instance-origin-port>`	The port of the host that contains data to be migrated.	`6379` once the developer will use the Reserve Tunnel	Yes
`<a9s-instance-origin-password>`	The password of the host that contains data to be migrated.	---	Yes
`<a9s-instance-origin-username>`	The username of the host that contains data to be migrated.	---	Yes
`<a9s-instance-origin-logical-database-number>`	The database logical number. Default: 0	0	Yes
`<origin_ca_cert_path>`	The CA path when the Service Instance is SSL. Can be obtained through a service-binding or service-key.	---	No
`<a9s-instance-destination-host>`	The domain or ip of the host that the data will be migrated to.	---	Yes
`<a9s-instance-destination-port>`	The port of the host that the data will be migrated to.	`6379`	Yes
`<a9s-instance-destination-password>`	The password of the host that the data will be migrated to.	---	Yes
`<a9s-instance-destination-username>`	The username of the host that the data will be migrated to.	---	Yes
`<destination_ca_cert_path>`	The CA path when the Service Instance is SSL. Can be obtained through a service-binding or service-key.	---	No
`<timeout>`	The timeout of each MIGRATION operation.	`5000`	Yes

Retrieving The Destination Service Instance Information

To get the necessary information from your destination Service Instance you can create a service key.

cf create-service-key <cf-instance-destination-service-instance> key

This key will contain all the necessary values, which you can fetch directly via the grep command as follows:

cf service-key <cf-instance-destination-service-instance> key | grep '"host"' # The instance destination host domain (`<a9s-instance-destination-host>`)
cf service-key <cf-instance-destination-service-instance> key | grep '"port"' #The instance destination port (`<a9s-instance-destination-port>`)
cf service-key <cf-instance-destination-service-instance> key | grep '"password"' # The instance destination password (`<a9s-instance-destination-password>`)
cf service-key <cf-instance-destination-service-instance> key | grep '"username"' # The instance destination password (`<a9s-instance-destination-password>`)

Migrate to a9s KeyValue Using the Disaster Recovery Feature​

Migrate to a9s KeyValue Manually​

Prerequisites​

Known Issues​

Before Migrating​

Isolating The Nodes Before The Migration​

Create a New a9s KeyValue Destination Instance​

Create an SSH Tunnel​

Migrate to a9s KeyValue Using the MIGRATE Option​

Perform the Migration​

Check The Database Consistency​

Check the Output of MIGRATE Requests at migration-output-checker.txt​

Migrate Using Manual DUMP & RESTORE​

Perform The Migration - Non-Binary-Safe Script​

Perform The Migration - Binary-Safe Script​

Checking For Migration Errors​

Check The Database Consistency​

Check the Script's Output at migrate-manual-dump-restore.txt​

Check The Database Consistency (General Approach)​

Compare Data Between the Service Instances​

Compare Keys Between the Service Instances​

Placeholder Values​

Retrieving The Destination Service Instance Information​

Migrate to a9s KeyValue Using the Disaster Recovery Feature

Migrate to a9s KeyValue Manually

Prerequisites

Known Issues

Before Migrating

Isolating The Nodes Before The Migration

Create a New a9s KeyValue Destination Instance

Create an SSH Tunnel

Migrate to a9s KeyValue Using the MIGRATE Option

Perform the Migration

Check The Database Consistency

Check the Output of MIGRATE Requests at `migration-output-checker.txt`

Migrate Using Manual DUMP & RESTORE

Perform The Migration - Non-Binary-Safe Script

Perform The Migration - Binary-Safe Script

Checking For Migration Errors

Check The Database Consistency

Check the Script's Output at `migrate-manual-dump-restore.txt`

Check The Database Consistency (General Approach)

Compare Data Between the Service Instances

Compare Keys Between the Service Instances

Placeholder Values

Retrieving The Destination Service Instance Information