Skip to main content
Version: 55.0.0

Migration

This document describes how to migrate data from an older a9s Redis version to a newer one. We present two approaches, the first one uses the MIGRATE command, while the other one performs an element-wise DUMP & RESTORE.

The following migration paths have been tested and are supported by the a9s Data Services Framework:

  • a9s Redis 6 -> a9s Redis 7 GA
  • a9s Redis 7 RC -> a9s Redis 7 GA (Special Case)
a9s Redis 7 RC Special Case

Please be aware that a difference exists between Redis 7 RC and Redis 7 GA due to the implementation of Redis ACL in Redis 7 GA. This results in a minor alteration in the migration process. This discrepancy will be highlighted during the subsequent steps.

Prerequisites

Known Issues

  • Downtime during the migration. It will depend on the amount of data inside your Redis service instance and network connection throughput between both Redis service instances.
  • Every scenario in this migration was tested with Redis instances following best practices for key naming. So, it is known that there may be problems when trying to migrate that when key names do not follow best practices. An example of this limitation is the use of keys with names containing binary content or other special characters can potentially cause issues during the migration. Please note that you might need to perform individual migration for any existing special keys, although we do provide an alternative Ruby-based solution in Migrate Redis Using Manual DUMP & RESTORE for such cases.
  • If a Redis service instance uses a logical database, in order to migrate the whole Redis content, it is necessary to migrate the Redis logical database one after another. A piece of information about logical databases:

    Out of the box, a Redis instance supports 16 logical databases. These databases are effectively siloed off from one another, and when you run a command in one database, it doesn't affect any of the data stored in other databases in your Redis instance.

Before Migrating

It is highly recommended to isolate the Redis service instance before starting the migration. This avoids potential data inconsistencies that could be caused if the Redis service instance is receiving new data while processing the migration at the same time.

Isolating The Nodes Before The Migration

Overall the goal is to avoid client connections. That way, we ensure that we are not writing data into the a9s Redis service instance during the migration process.

This can be achieved by unbinding all applications from your Redis service instance.

Save the RDB File (Redis Dump File)

We recommend saving your service instance's RDB file as failsafe in the event of unexpected data loss. This file must be stored safely, at least until the consistency of the database has been confirmed, post-migration.

You can download the latest backup from a9s Service Dashboard.

Save the RDB File (Alternative)
Alternatively, it is possible to download the RDB file for a specific service instance using `redis-cli`:
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli --user <a9s-redis-origin-username> -h <a9s-redis-origin-host> -p 6379 \
-a <a9s-redis-origin-password> ${origin_tls:-} --rdb /tmp/dump.rdb
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p 6379 -a <a9s-redis-origin-password> ${origin_tls:-} \
--rdb /tmp/dump.rdb

Note:

  • For a9s Redis 7 GA version or newer, this alternative does not work.

Please refer to the Placeholder Values table at the end of this guide for further details on <a9s-redis-origin-host>, <a9s-redis-origin-username>, <a9s-redis-origin-password>, <origin_ca_cert_path> and similar placeholders on this document.

Create New a9s Redis destination Instance

Create a new and empty a9s Redis destination service instance that will be the target for the data migration from an existing Redis origin service instance.

Note:

  • Do not insert any data inside of the machine before the migration and checking are done.
  • Create a service instance with the same capabilities or greater.

Reverse Tunnel

The goal is to give access to the application developer to the Redis service instance. And for this, it is necessary to make a SSH tunnels to the Redis origin service instance.

In the end, we want to achieve the following scenario below.

                    (port 6379)                             (port 6379)
/----> * Redis Origin * ----- Redis MIGRATE -----> * Redis Destination *
/
* CF Application * (via SSH Tunnel) [infrastructure network]
--------/-----------------------------------------------------------------
/ [Developer network]
* Developer Machine * (local port 6379)

Create Tunnel

It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the service for any purpose such as debugging. Cloud Foundry (CF) provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.

First of all you must have an application bound to the service. How to do this see Bind an application to a Service Instance.

Note: cf ssh support must be enabled in the platform. Ask your platform operator if you are not sure.

Follow the section Create a Tunnel to The Service to create the reverse tunnel.

Migrate Redis Using the MIGRATE Option

Warning Notes
  • Migrating between an SSL service instance and a non-SSL service instance, or vice versa, is not supported by Redis.

  • When migrating data between SSL service instances they MUST have the same CA, or it MUST be possible to verify the certificate of the destination with the CA of the origin service instance. Otherwise, it will fail because the TLS connection can not be verified.

For all of the cases above, please use the Migrate Redis Using Manual DUMP & RESTORE.

The main strategy to migrate the whole dataset from one Redis service instance to another is based on the Redis MIGRATE feature.

This feature comes in the form of a simple command that can be run through redis-cli. Using the CLI, we shall connect to the existing instance, and execute the MIGRATE command. The command accepts the hostname and credentials for the new instance, which implies that there will be a direct connection between the two instances during the migration.

The vendor documentation of the feature also says:

The command is atomic and blocks the two instances for the time required to transfer the key, at any given time the key will appear to exist in a given instance or the other instance unless a timeout error occurs.

Hence, there will be a system downtime during the migration. Thus it is suggested to trigger the migration process during an appropriate time window; likely in the moment in the week, the day, and the period with less impact for the client of this specific Redis service instance. The application developer must decide which timeframe fits these criteria.

Migration Duration

Based on internal and isolated tests, migrating 1GB takes about 50 seconds on average. However, it will depend on each environment and its particularities. Be aware that it might take some time, so it is necessary to plan wisely.

Perform the Migration

Use the redis-cli to retrieve the keys and migrate between the Redis service instances. Replace all placeholder values in the command below and execute it afterwards. These commands may change depending on the version of Redis, make sure to run the appropriate command for your case. This command is valid for Redis 7 GA and above:

# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

redis_cli_origin_command="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> -a <a9s-redis-origin-password> -n <a9s-redis-origin-logical-database-number> ${origin_tls:-}"

${redis_cli_origin_command} --raw KEYS '*' | xargs ${redis_cli_origin_command} MIGRATE \
<a9s-redis-destination-host> <a9s-redis-destination-port> "" <a9s-redis-origin-logical-database-number> <timeout> \
COPY AUTH2 <a9s-redis-destination-username> <a9s-redis-destination-password> KEYS > /tmp/migration-output-checker.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

redis_cli_origin_command="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> -a <a9s-redis-origin-password> -n <a9s-redis-origin-logical-database-number> ${origin_tls:-}"

${redis_cli_origin_command} --raw KEYS '*' | xargs ${redis_cli_origin_command} MIGRATE \
<a9s-redis-destination-host> <a9s-redis-destination-port> "" <a9s-redis-origin-logical-database-number> <timeout> \
COPY AUTH <a9s-redis-destination-password> KEYS > /tmp/migration-output-checker.txt

The migration-output-checker.txt contains the output of the MIGRATE command, and it is used in a further step to make sure the migration was executed with success. For more information see Check The Database Consistency.

The command above can only be executed by an application developer once they have access to the Redis service instance via the reverse tunnel.

Furthermore, this command is composed of multiple commands, all of which serve a specific function:

  • The first command, redis-cli --raw KEYS '*', lists all the keys stored in the existing instance.
  • The second command, xargs redis-cli MIGRATE <destination-host> <destination-port> "" 0 5000 COPY AUTH <pass> KEYS, uses the output of the first command to MIGRATE the data of the source host to the destination host.
    • The part following the first command and preceding the second one simply utilizes the tunnel to connect to the existing instance, where the second command is executed.
  • The COPY argument must be present in the command to avoid deleting the key from the source host.

Please refer to the Placeholder Values table at the end of this guide for more specific details on the placeholders.

Logical Databases

If you are using a logical database, and you have more than one, this command must be executed as many times as the amount of the logical databases. It is necessary to change the <a9s-redis-original-logical-database-number> value accordingly.

Key Names

If the origin Redis instance contains keys with names containing special characters this migration process will not work. For example, if one key contains a breakline character this can be a problem.

Check The Database Consistency

Check the database consistency between the original Redis service instance and the new Redis service instance databases following the steps below.

Also check the consistency using the Check The Database Consistency (General Approach).

Check the Output of MIGRATE Requests at migration-output-checker.txt.

The migration-output-checker.txt content is the output result of each MIGRATE request made to the Redis instance. It must contain only "OK".

Check the content:

cat /tmp/migration-output-checker.txt | grep --invert-match "OK"

The output of this command must be empty, otherwise it will return an error message with details on what went wrong during the migration. Below we list some of the potential error messages:

Error MessagePossible CauseTroubleshooting
NOKEYThe key does not exist in the source Redis service instance.Double check if the first part of the command (redis-cli --raw KEYS '*') is correct. The output of this command must have all keys present in the Redis instance.
IOERR error or timeout connecting to the clientIt is likely a communication problem between the two Redis instances.Double check if the source Redis service instance has access to the destination Redis service instance.
ERR Target instance replied with error: ERR invalid passwordAuthentication does not work.Double check if the password is correct.

Migrate Redis Using Manual DUMP & RESTORE

In case the method using MIGRATE does not work, we can make use of what actually takes place behind the scenes in the MIGRATE command, albeit without assured atomicity and a longer duration:

  1. DUMP - every key in the existing instance is extracted using this command
  2. RESTORE - every key (with data) is reinstated in the new instance

As in the previous examples, for the following scripts you need to make sure to replace the given placeholders (see table below) with the actual values to create tunneled connections to the a9s Redis Instances. We provide two alternatives, the first one being a BASH script while the second one uses Ruby instead. Due to BASH's non-binary-safe nature, we provide the Ruby script, as it can handle the migration of certain binary values or keys that the first script cannot.

Perform The Migration - Non-Binary-Safe Script

The following script (migrate_manual_dump_restore.sh) will handle this task.

Version Warning

Please, note that if an a9s Redis 7 RC version or older is either the origin or destination, the --user <a9s-redis-origin-username> parameter MUST NOT be passed in the commands.

#!/bin/bash

# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
export redis_origin_instance="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> -a <a9s-redis-origin-password> --user <a9s-redis-origin-username> ${origin_tls:-}"

# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
export redis_destination_instance="redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> -a <a9s-redis-destination-password> --user <a9s-redis-destination-username> ${destination_tls:-}"

for key in $(${redis_origin_instance} --scan); do
# For Linux, use `head -c-1` instead of `ghead -c-1` in the command below.
${redis_origin_instance} --raw DUMP "${key}" 2> /dev/null | ghead -c-1 > /tmp/redis_dump
ttl=$(${redis_origin_instance} --raw TTL "${key}" 2> /dev/null)
case ${ttl} in
-2)
echo "Key ${key} not found in origin instance.. Skipping."
;;
-1)
${redis_destination_instance} DEL "${key}" > /dev/null 2>&1
restore_output=$(cat /tmp/redis_dump | ${redis_destination_instance} --no-auth-warning -x RESTORE "${key}" 0)
[ "${restore_output}" != "OK" ] && \
echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue

echo "${key} migrated successfully!"
;;
*)
${redis_destination_instance} DEL "${key}" > /dev/null 2>&1
restore_output=$(cat /tmp/redis_dump | ${redis_destination_instance} --no-auth-warning -x RESTORE "${key}" "${ttl}")
[ "${restore_output}" != "OK" ] && \
echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue

echo "${key} [TTL = ${ttl}] migrated successfully!"
;;
esac
done

Execute the script:

bash migrate_manual_dump_restore.sh > "/tmp/migrate-manual-dump-restore.txt"
Important Details
  • The script above uses a third command called TTL. This command reads the Time-to-Live specification of a key in your existing instance. Using this command we make sure to ensure the original expiration time is respected in the new instance.

  • The script relies on a shell utility called head. If you are using a MacOS environment, you need to install its GNU-based alternative called ghead.

  • Since this is a manual migration, it is strongly suggested that you isolate the nodes, see Isolating The Nodes Before The Migration for more information. This is necessary, because unlike the MIGRATE command, this script does not ensure atomicity.

  • Due to the details mentioned above, this script should only be used as a last resort.

Perform The Migration - Binary-Safe Script

The migrate_manual_dump_restore.sh script in the previous section is not able to correctly handle cases of Keys and/or Values having a binary structure. That is because the script will normally store the Keys and Values in variables in a non-binary-safe manner. This becomes an issue when those values are then provided to the redis-cli in commands like RESTORE or even MIGRATE, which we describe above. To address these issues, we provide a Ruby script below which uses the DUMP & RESTORE idea while being binary-safe. We will use the name migrate_manual_dump_restore.rb.

require 'redis'

origin_instance = Redis.new(host: <a9s-redis-origin-host>, port: <a9s-redis-origin-port>, db: <a9s-redis-origin-logical-database-number>, password: <a9s-redis-origin-password>)
# Example when using SSH tunnel:
# origin_instance = Redis.new(host: 'localhost', port: 6379, db: 0, password: 'a9sa860dc0ef50586d8fe80dcc4db7633672ccccccccc91db1e24fdfd0b59d')

destination_instance = Redis.new(host: <a9s-redis-destination-host>, port: <a9s-redis-destination-port>, db: <a9s-redis-origin-logical-database-number>, password: <a9s-redis-destination-password>)
# Example when using SSH tunnel:
# destination_instance = Redis.new(host: 'localhost', port: 6380, db: 0, password: 'a9sd172addf8c320b20961daaaeb11cfd4d5f3656adc11b75gr6144e07547e5')

origin_instance.scan_each { |key|
begin

ttl = origin_instance.ttl(key)

if ttl == -2 then
next
elsif ttl == -1 then
ttl = 0
else
ttl = 1000 * ttl
end

dumped_value = source_instance.dump(key)
destination_instance.restore(key, ttl, dumped_value, :replace => true)

rescue Redis::CommandError => e
puts "Migration ERROR restoring key: #{key} - #{e.message}"
end
}

source_instance.quit
destination_instance.quit

In order to execute the script above, you need to have Ruby 3.x installed in your local environment. You also need to install a gem called redis, by running gem install redis. The gem utility should be present in your environment once you install Ruby. Once you have the local environment setup, you need to create the SSH tunnels as described at the beginning of this guide in Create Tunnel.

Then you need to use the correct Placeholder Values and execute the following command in the command-line:

ruby migrate_manual_dump_restore.rb > /tmp/migrate-manual-dump-restore.txt
Important Details

Checking For Migration Errors

The migrate-manual-dump-restore.txt contains the output of the execution of any of the scripts above, and it is used in the next step to make sure the migration was executed with success.

Check The Database Consistency

Check the database consistency between the original Redis service instance and the new Redis service instance databases following the steps below.

Also check the consistency using the Check The Database Consistency (General Approach).

Check the Script's Output at migrate-manual-dump-restore.txt

The migrate-manual-dump-restore.txt content is the output result of migrate_manual_dump_restore.sh script. It MUST not contain errors.

Check the content:

cat /tmp/migrate-manual-dump-restore.txt | grep "Migration ERROR"

Should a problem occur at any point of process, this might indicate that the migration did not finish successfully, in this case make sure you save a copy of the migrate-manual-dump-restore.txt for further investigation and contact your platform operator.

Check The Database Consistency (General Approach)

Check the database consistency between the original Redis service instance and the new Redis service instance databases.

Compare Data Between the Redis Service Instances

Limitations

If by any chance the origin Redis service instance was not isolated and kept receiving new data during the migration the following check will not work.

  • Execute the info keyspace redis-cli command under each host. The keys and expires values must be equal. This command should only be run on a9s Redis 7 GA and above:
# Redis Origin

# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} info keyspace

# Redis Destination

# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} info keyspace
a9s Redis 7 RC and below
# Redis Origin

# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> \
-a <a9s-redis-origin-password> ${origin_tls:-} info keyspace

# Redis Destination

# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> \
-a <a9s-redis-destination-password> ${destination_tls:-} info keyspace
  • Execute the info memory redis-cli command under each host. The values must be similar with only a few kilobytes of difference, if any.
# Redis Origin

# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} info memory | grep "used_memory:\|used_memory_human"

# Redis Destination

# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} info memory | grep "used_memory:\|used_memory_human"
a9s Redis 7 RC and below
# Redis Origin

# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> \
-a <a9s-redis-origin-password> ${origin_tls:-} info memory | grep "used_memory:\|used_memory_human"

# Redis Destination

# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> \
-a <a9s-redis-destination-password> ${destination_tls:-} info memory | grep "used_memory:\|used_memory_human"
note

In the case of migrations from a9s Redis 6 or a9s Redis 7 RC to Redis 7 GA and vice-versa, please pay attention to use the appropriate commands for the versions of the origin and destination instances.

Compare Keys Between the Redis Service Instances

This step will compare all keys between both Redis instances.

  • Get all source Redis service instance keys and sort them.
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} --raw keys '*' > /tmp/redis-origin.txt
sort --parallel=2 -uo /tmp/redis-origin-sorted.txt /tmp/redis-origin.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"

redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> -a <a9s-redis-origin-password> ${origin_tls:-} \
--raw keys '*' > /tmp/redis-origin.txt
sort --parallel=2 -uo /tmp/redis-origin-sorted.txt /tmp/redis-origin.txt
  • Get all destination Redis service instance keys and sort them.
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"

redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} --raw keys '*' > /tmp/redis-destination.txt
sort --parallel=2 -uo /tmp/redis-destination-sorted.txt /tmp/redis-destination.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"

redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> -a <a9s-redis-destination-password> \
${destination_tls:-} --raw keys '*' > /tmp/redis-destination.txt
sort --parallel=2 -uo /tmp/redis-destination-sorted.txt /tmp/redis-destination.txt
  • Compare both Redis service instances keys
diff /tmp/redis-origin-sorted.txt /tmp/redis-destination-sorted.txt

The output MUST NOT have differences.

Placeholder Values

PlaceholderDescriptionExampleRequired?
<a9s-redis-origin-host>The domain or ip of the host that contains data to be migrated.localhost or 127.0.0.1Yes
<a9s-redis-origin-port>The port of the host that contains data to be migrated.6379 once the developer will use the Reserve TunnelYes
<a9s-redis-origin-password>The password of the host that contains data to be migrated.---Yes
<a9s-redis-origin-username>The username of the host that contains data to be migrated.---Yes for a9s Redis 7 GA and above
<a9s-redis-origin-logical-database-number>The Redis database logical number. Default: 00Yes
<origin_ca_cert_path>The CA path when the Service Instance is SSL. Can be obtained through a service-binding or service-key.---No
<a9s-redis-destination-host>The domain or ip of the host that the data will be migrated to.---Yes
<a9s-redis-destination-port>The port of the host that the data will be migrated to.6379Yes
<a9s-redis-destination-password>The password of the host that the data will be migrated to.---Yes
<a9s-redis-destination-username>The username of the host that the data will be migrated to.---Yes for a9s Redis 7 GA and above
<destination_ca_cert_path>The CA path when the Service Instance is SSL. Can be obtained through a service-binding or service-key.---No
<timeout>The timeout of each Redis MIGRATION operation.5000Yes

Retrieving The Destination Redis Information

To get the necessary information from your destination Redis service instance you can create a service key.

cf create-service-key <cf-redis-destination-service-instance> key

This key will contain all the necessary values, which you can fetch directly via the grep command as follows:

cf service-key <cf-redis-destination-service-instance> key | grep '"host"' # The Redis destination host domain (`<a9s-redis-destination-host>`)
cf service-key <cf-redis-destination-service-instance> key | grep '"port"' #The Redis destination port (`<a9s-redis-destination-port>`)
cf service-key <cf-redis-destination-service-instance> key | grep '"password"' # The Redis destination password (`<a9s-redis-destination-password>`)

# Redis 7 GA and above
cf service-key <cf-redis-destination-service-instance> key | grep '"username"' # The Redis destination password (`<a9s-redis-destination-password>`)