Migration
This document describes how to migrate data from an older a9s Redis version to a newer one. We present two approaches,
the first one uses the MIGRATE
command, while the other one performs an element-wise DUMP
& RESTORE
.
The following migration paths have been tested and are supported by the a9s Data Services Framework:
- a9s Redis 3 -> a9s Redis 4
- a9s Redis 4 -> a9s Redis 5
- a9s Redis 5 -> a9s Redis 6
- a9s Redis 6 -> a9s Redis 7 GA
- a9s Redis 7 RC -> a9s Redis 7 GA (Special Case)
Please be aware that a difference exists between Redis 7 RC and Redis 7 GA due to the implementation of Redis ACL in Redis 7 GA. This results in a minor alteration in the migration process. This discrepancy will be highlighted during the subsequent steps.
Prerequisites
- Install redis-cli.
Known Issues
- Downtime during the migration. It will depend on the amount of data inside your Redis service instance and network connection throughput between both Redis service instances.
- Every scenario in this migration was tested with Redis instances following best practices for key naming. So, it is known that there may be problems when trying to migrate that when key names do not follow best practices. An example of this limitation is the use of keys with names containing binary content or other special characters can potentially cause issues during the migration. Please note that you might need to perform individual migration for any existing special keys, although we do provide an alternative Ruby-based solution in Migrate Redis Using Manual DUMP & RESTORE for such cases.
- If a Redis service instance uses a logical database, in order to migrate the whole Redis content, it is necessary to
migrate the Redis logical database one after another. A piece of information about logical databases:
Out of the box, a Redis instance supports 16 logical databases. These databases are effectively siloed off from one another, and when you run a command in one database, it doesn't affect any of the data stored in other databases in your Redis instance.
Before Migrating
It is highly recommended to isolate the Redis service instance before starting the migration. This avoids potential data inconsistencies that could be caused if the Redis service instance is receiving new data while processing the migration at the same time.
Isolating The Nodes Before The Migration
Overall the goal is to avoid client connections. That way, we ensure that we are not writing data into the a9s Redis service instance during the migration process.
This can be achieved by unbinding all applications from your Redis service instance.
We recommend saving your service instance's RDB file as failsafe in the event of unexpected data loss. This file must be stored safely, at least until the consistency of the database has been confirmed, post-migration.
You can download the latest backup from a9s Service Dashboard.
Save the RDB File (Alternative)
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli --user <a9s-redis-origin-username> -h <a9s-redis-origin-host> -p 6379 \
-a <a9s-redis-origin-password> ${origin_tls:-} --rdb /tmp/dump.rdb
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p 6379 -a <a9s-redis-origin-password> ${origin_tls:-} \
--rdb /tmp/dump.rdb
Note:
- For a9s Redis 7 GA version or newer, this alternative does not work.
Please refer to the Placeholder Values table at the end of this guide for further details on <a9s-redis-origin-host>
,
<a9s-redis-origin-username>
, <a9s-redis-origin-password>
, <origin_ca_cert_path>
and similar placeholders on this
document.
Create New a9s Redis destination Instance
Create a new and empty a9s Redis destination service instance that will be the target for the data migration from an existing Redis origin service instance.
Note:
- Do not insert any data inside of the machine before the migration and checking are done.
- Create a service instance with the same capabilities or greater.
Reverse Tunnel
The goal is to give access to the application developer to the Redis service instance. And for this, it is necessary to make a SSH tunnels to the Redis origin service instance.
In the end, we want to achieve the following scenario below.
(port 6379) (port 6379)
/----> * Redis Origin * ----- Redis MIGRATE -----> * Redis Destination *
/
* CF Application * (via SSH Tunnel) [infrastructure network]
--------/-----------------------------------------------------------------
/ [Developer network]
* Developer Machine * (local port 6379)
Create Tunnel
It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the service for any purpose such as debugging. Cloud Foundry (CF) provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.
First of all you must have an application bound to the service. How to do this see Bind an application to a Service Instance.
Note: cf ssh
support must be enabled in the platform. Ask your platform operator if you are not sure.
Follow the section Create a Tunnel to The Service to create the reverse tunnel.
Migrate Redis Using the MIGRATE Option
Migrating between an SSL service instance and a non-SSL service instance, or vice versa, is not supported by Redis.
When migrating data between SSL service instances they MUST have the same CA, or it MUST be possible to verify the certificate of the destination with the CA of the origin service instance. Otherwise, it will fail because the TLS connection can not be verified.
For all of the cases above, please use the Migrate Redis Using Manual DUMP & RESTORE.
The main strategy to migrate the whole dataset from one Redis service instance to another is based on the Redis MIGRATE feature.
This feature comes in the form of a simple command that can be run through redis-cli
. Using the CLI, we shall connect
to the existing instance, and execute the MIGRATE
command. The command accepts the hostname and credentials for the
new instance, which implies that there will be a direct connection between the two instances during
the migration.
The vendor documentation of the feature also says:
The command is atomic and blocks the two instances for the time required to transfer the key, at any given time the key will appear to exist in a given instance or the other instance unless a timeout error occurs.
Hence, there will be a system downtime during the migration. Thus it is suggested to trigger the migration process during an appropriate time window; likely in the moment in the week, the day, and the period with less impact for the client of this specific Redis service instance. The application developer must decide which timeframe fits these criteria.
Based on internal and isolated tests, migrating 1GB takes about 50 seconds on average. However, it will depend on each environment and its particularities. Be aware that it might take some time, so it is necessary to plan wisely.
Perform the Migration
Use the redis-cli
to retrieve the keys and migrate between the Redis service instances. Replace all placeholder values
in the command below and execute it afterwards. These commands may change depending on the version of Redis, make sure
to run the appropriate command for your case. This command is valid for Redis 7 GA and above:
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis_cli_origin_command="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> -a <a9s-redis-origin-password> -n <a9s-redis-origin-logical-database-number> ${origin_tls:-}"
${redis_cli_origin_command} --raw KEYS '*' | xargs ${redis_cli_origin_command} MIGRATE \
<a9s-redis-destination-host> <a9s-redis-destination-port> "" <a9s-redis-origin-logical-database-number> <timeout> \
COPY AUTH2 <a9s-redis-destination-username> <a9s-redis-destination-password> KEYS > /tmp/migration-output-checker.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis_cli_origin_command="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> -a <a9s-redis-origin-password> -n <a9s-redis-origin-logical-database-number> ${origin_tls:-}"
${redis_cli_origin_command} --raw KEYS '*' | xargs ${redis_cli_origin_command} MIGRATE \
<a9s-redis-destination-host> <a9s-redis-destination-port> "" <a9s-redis-origin-logical-database-number> <timeout> \
COPY AUTH <a9s-redis-destination-password> KEYS > /tmp/migration-output-checker.txt
The migration-output-checker.txt
contains the output of the MIGRATE
command, and it is used in a further step to
make sure the migration was executed with success. For more information see
Check The Database Consistency.
The command above can only be executed by an application developer once they have access to the Redis service instance via the reverse tunnel.
Furthermore, this command is composed of multiple commands, all of which serve a specific function:
- The first command,
redis-cli --raw KEYS '*'
, lists all the keys stored in the existing instance. - The second command,
xargs redis-cli MIGRATE <destination-host> <destination-port> "" 0 5000 COPY AUTH <pass> KEYS
, uses the output of the first command toMIGRATE
the data of the source host to the destination host.- The part following the first command and preceding the second one simply utilizes the tunnel to connect to the existing instance, where the second command is executed.
- The
COPY
argument must be present in the command to avoid deleting the key from the source host.
Please refer to the Placeholder Values table at the end of this guide for more specific details on the placeholders.
If you are using a logical database, and you have more than one, this command must be executed as many times as the
amount of the logical databases. It is necessary to change the <a9s-redis-original-logical-database-number>
value
accordingly.
If the origin Redis instance contains keys with names containing special characters this migration process will not work. For example, if one key contains a breakline character this can be a problem.
Check The Database Consistency
Check the database consistency between the original Redis service instance and the new Redis service instance databases following the steps below.
Also check the consistency using the Check The Database Consistency (General Approach).
Check the Output of MIGRATE Requests at migration-output-checker.txt
.
The migration-output-checker.txt
content is the output result of each MIGRATE
request made to the Redis instance. It
must contain only "OK".
Check the content:
cat /tmp/migration-output-checker.txt | grep --invert-match "OK"
The output of this command must be empty, otherwise it will return an error message with details on what went wrong during the migration. Below we list some of the potential error messages:
Error Message | Possible Cause | Troubleshooting |
---|---|---|
NOKEY | The key does not exist in the source Redis service instance. | Double check if the first part of the command (redis-cli --raw KEYS '*' ) is correct. The output of this command must have all keys present in the Redis instance. |
IOERR error or timeout connecting to the client | It is likely a communication problem between the two Redis instances. | Double check if the source Redis service instance has access to the destination Redis service instance. |
ERR Target instance replied with error: ERR invalid password | Authentication does not work. | Double check if the password is correct. |
Migrate Redis Using Manual DUMP & RESTORE
In case the method using MIGRATE
does not work, we can make use of what actually takes place behind the scenes in the
MIGRATE
command, albeit without assured atomicity and a longer duration:
DUMP
- every key in the existing instance is extracted using this commandRESTORE
- every key (with data) is reinstated in the new instance
As in the previous examples, for the following scripts you need to make sure to replace the given placeholders (see table below) with the actual values to create tunneled connections to the a9s Redis Instances. We provide two alternatives, the first one being a BASH script while the second one uses Ruby instead. Due to BASH's non-binary-safe nature, we provide the Ruby script, as it can handle the migration of certain binary values or keys that the first script cannot.
Perform The Migration - Non-Binary-Safe Script
The following script (migrate_manual_dump_restore.sh
) will handle this task.
Please, note that if an a9s Redis 7 RC version or older is either the origin or destination, the
--user <a9s-redis-origin-username>
parameter MUST NOT be passed in the commands.
#!/bin/bash
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
export redis_origin_instance="redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> -a <a9s-redis-origin-password> --user <a9s-redis-origin-username> ${origin_tls:-}"
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
export redis_destination_instance="redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> -a <a9s-redis-destination-password> --user <a9s-redis-destination-username> ${destination_tls:-}"
for key in $(${redis_origin_instance} --scan); do
# For Linux, use `head -c-1` instead of `ghead -c-1` in the command below.
${redis_origin_instance} --raw DUMP "${key}" 2> /dev/null | ghead -c-1 > /tmp/redis_dump
ttl=$(${redis_origin_instance} --raw TTL "${key}" 2> /dev/null)
case ${ttl} in
-2)
echo "Key ${key} not found in origin instance.. Skipping."
;;
-1)
${redis_destination_instance} DEL "${key}" > /dev/null 2>&1
restore_output=$(cat /tmp/redis_dump | ${redis_destination_instance} --no-auth-warning -x RESTORE "${key}" 0)
[ "${restore_output}" != "OK" ] && \
echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue
echo "${key} migrated successfully!"
;;
*)
${redis_destination_instance} DEL "${key}" > /dev/null 2>&1
restore_output=$(cat /tmp/redis_dump | ${redis_destination_instance} --no-auth-warning -x RESTORE "${key}" "${ttl}")
[ "${restore_output}" != "OK" ] && \
echo "(Migration ERROR): ${key} failed to be migrated! Error: *${restore_output}*" && continue
echo "${key} [TTL = ${ttl}] migrated successfully!"
;;
esac
done
Execute the script:
bash migrate_manual_dump_restore.sh > "/tmp/migrate-manual-dump-restore.txt"
The script above uses a third command called TTL. This command reads the Time-to-Live specification of a key in your existing instance. Using this command we make sure to ensure the original expiration time is respected in the new instance.
The script relies on a shell utility called
head
. If you are using a MacOS environment, you need to install its GNU-based alternative calledghead
.Since this is a manual migration, it is strongly suggested that you isolate the nodes, see Isolating The Nodes Before The Migration for more information. This is necessary, because unlike the
MIGRATE
command, this script does not ensure atomicity.Due to the details mentioned above, this script should only be used as a last resort.
Perform The Migration - Binary-Safe Script
The migrate_manual_dump_restore.sh
script in the previous section is not able to correctly handle cases of Keys
and/or Values having a binary structure. That is because the script will normally store the Keys and Values in variables
in a non-binary-safe manner. This becomes an issue when those values are then provided to the redis-cli
in commands like
RESTORE
or even MIGRATE
, which we describe above. To address these issues, we provide a Ruby script below which uses
the DUMP & RESTORE
idea while being binary-safe. We will use the name migrate_manual_dump_restore.rb
.
require 'redis'
origin_instance = Redis.new(host: <a9s-redis-origin-host>, port: <a9s-redis-origin-port>, db: <a9s-redis-origin-logical-database-number>, password: <a9s-redis-origin-password>)
# Example when using SSH tunnel:
# origin_instance = Redis.new(host: 'localhost', port: 6379, db: 0, password: 'a9sa860dc0ef50586d8fe80dcc4db7633672ccccccccc91db1e24fdfd0b59d')
destination_instance = Redis.new(host: <a9s-redis-destination-host>, port: <a9s-redis-destination-port>, db: <a9s-redis-origin-logical-database-number>, password: <a9s-redis-destination-password>)
# Example when using SSH tunnel:
# destination_instance = Redis.new(host: 'localhost', port: 6380, db: 0, password: 'a9sd172addf8c320b20961daaaeb11cfd4d5f3656adc11b75gr6144e07547e5')
origin_instance.scan_each { |key|
begin
ttl = origin_instance.ttl(key)
if ttl == -2 then
next
elsif ttl == -1 then
ttl = 0
else
ttl = 1000 * ttl
end
dumped_value = source_instance.dump(key)
destination_instance.restore(key, ttl, dumped_value, :replace => true)
rescue Redis::CommandError => e
puts "Migration ERROR restoring key: #{key} - #{e.message}"
end
}
source_instance.quit
destination_instance.quit
In order to execute the script above, you need to have Ruby 3.x
installed in your local environment. You also need to install a gem
called redis
, by running gem install redis
. The gem
utility should be present in your environment once you install Ruby
. Once you have the local environment setup, you need to create the SSH tunnels as described at the beginning of this guide in Create Tunnel.
Then you need to use the correct Placeholder Values and execute the following command in the command-line:
ruby migrate_manual_dump_restore.rb > /tmp/migrate-manual-dump-restore.txt
The script above uses the same logic as the BASH script in Perform The Migration - Non-Binary-Safe Script, so atomicity is again not guaranteed.
Similarly, you are strongly advised to consider Isolating The Nodes Before The Migration before proceeding.
Unless an a9s Redis 7 RC version or older is either the origin or destination, the
--user <a9s-redis-origin-username>
parameter has to be included in the commands.
Checking For Migration Errors
The migrate-manual-dump-restore.txt
contains the output of the execution of any of the scripts above,
and it is used in the next step to make sure the migration was executed with success.
Check The Database Consistency
Check the database consistency between the original Redis service instance and the new Redis service instance databases following the steps below.
Also check the consistency using the Check The Database Consistency (General Approach).
Check the Script's Output at migrate-manual-dump-restore.txt
The migrate-manual-dump-restore.txt
content is the output result of migrate_manual_dump_restore.sh
script. It
MUST not contain errors.
Check the content:
cat /tmp/migrate-manual-dump-restore.txt | grep "Migration ERROR"
Should a problem occur at any point of process, this might indicate that the migration did not finish successfully, in
this case make sure you save a copy of the migrate-manual-dump-restore.txt
for further investigation and contact
your platform operator.
Check The Database Consistency (General Approach)
Check the database consistency between the original Redis service instance and the new Redis service instance databases.
Compare Data Between the Redis Service Instances
If by any chance the origin Redis service instance was not isolated and kept receiving new data during the migration the following check will not work.
- Execute the
info keyspace
redis-cli command under each host. Thekeys
andexpires
values must be equal. This command should only be run on a9s Redis 7 GA and above:
# Redis Origin
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} info keyspace
# Redis Destination
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} info keyspace
a9s Redis 7 RC and below
# Redis Origin
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> \
-a <a9s-redis-origin-password> ${origin_tls:-} info keyspace
# Redis Destination
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> \
-a <a9s-redis-destination-password> ${destination_tls:-} info keyspace
- Execute the
info memory
redis-cli command under each host. The values must be similar with only a few kilobytes of difference, if any.
# Redis Origin
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} info memory | grep "used_memory:\|used_memory_human"
# Redis Destination
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} info memory | grep "used_memory:\|used_memory_human"
a9s Redis 7 RC and below
# Redis Origin
# set `tls` value if the Origin service instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> \
-a <a9s-redis-origin-password> ${origin_tls:-} info memory | grep "used_memory:\|used_memory_human"
# Redis Destination
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> \
-a <a9s-redis-destination-password> ${destination_tls:-} info memory | grep "used_memory:\|used_memory_human"
In the case of migrations from a9s Redis 6 or a9s Redis 7 RC to Redis 7 GA and vice-versa, please pay attention to use the appropriate commands for the versions of the origin and destination instances.
Compare Keys Between the Redis Service Instances
This step will compare all keys between both Redis instances.
- Get all source Redis service instance keys and sort them.
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> --user <a9s-redis-origin-username> \
-a <a9s-redis-origin-password> ${origin_tls:-} --raw keys '*' > /tmp/redis-origin.txt
sort --parallel=2 -uo /tmp/redis-origin-sorted.txt /tmp/redis-origin.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"
redis-cli -h <a9s-redis-origin-host> -p <a9s-redis-origin-port> -a <a9s-redis-origin-password> ${origin_tls:-} \
--raw keys '*' > /tmp/redis-origin.txt
sort --parallel=2 -uo /tmp/redis-origin-sorted.txt /tmp/redis-origin.txt
- Get all destination Redis service instance keys and sort them.
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> --user <a9s-redis-destination-username> \
-a <a9s-redis-destination-password> ${destination_tls:-} --raw keys '*' > /tmp/redis-destination.txt
sort --parallel=2 -uo /tmp/redis-destination-sorted.txt /tmp/redis-destination.txt
a9s Redis 7 RC and below
# set `tls` value if the Origin service instance is SSL
# destination_tls="--tls --cacert <destination_ca_cert_path>"
redis-cli -h <a9s-redis-destination-host> -p <a9s-redis-destination-port> -a <a9s-redis-destination-password> \
${destination_tls:-} --raw keys '*' > /tmp/redis-destination.txt
sort --parallel=2 -uo /tmp/redis-destination-sorted.txt /tmp/redis-destination.txt
- Compare both Redis service instances keys
diff /tmp/redis-origin-sorted.txt /tmp/redis-destination-sorted.txt
The output MUST NOT have differences.
Placeholder Values
Placeholder | Description | Example | Required? |
---|---|---|---|
<a9s-redis-origin-host> | The domain or ip of the host that contains data to be migrated. | localhost or 127.0.0.1 | Yes |
<a9s-redis-origin-port> | The port of the host that contains data to be migrated. | 6379 once the developer will use the Reserve Tunnel | Yes |
<a9s-redis-origin-password> | The password of the host that contains data to be migrated. | --- | Yes |
<a9s-redis-origin-username> | The username of the host that contains data to be migrated. | --- | Yes for a9s Redis 7 GA and above |
<a9s-redis-origin-logical-database-number> | The Redis database logical number. Default: 0 | 0 | Yes |
<origin_ca_cert_path> | The CA path when the service instance is SSL. | --- | No |
<a9s-redis-destination-host> | The domain or ip of the host that the data will be migrated to. | --- | Yes |
<a9s-redis-destination-port> | The port of the host that the data will be migrated to. | 6379 | Yes |
<a9s-redis-destination-password> | The password of the host that the data will be migrated to. | --- | Yes |
<a9s-redis-destination-username> | The username of the host that the data will be migrated to. | --- | Yes for a9s Redis 7 GA and above |
<destination_ca_cert_path> | The CA path when the service instance is SSL. | --- | No |
<timeout> | The timeout of each Redis MIGRATION operation. | 5000 | Yes |
Retrieving The Destination Redis Information
To get the necessary information from your destination Redis service instance you can create a service key.
cf create-service-key <cf-redis-destination-service-instance> key
This key will contain all the necessary values, which you can fetch directly via the grep
command as follows:
cf service-key <cf-redis-destination-service-instance> key | grep '"host"' # The Redis destination host domain (`<a9s-redis-destination-host>`)
cf service-key <cf-redis-destination-service-instance> key | grep '"port"' #The Redis destination port (`<a9s-redis-destination-port>`)
cf service-key <cf-redis-destination-service-instance> key | grep '"password"' # The Redis destination password (`<a9s-redis-destination-password>`)
# Redis 7 GA and above
cf service-key <cf-redis-destination-service-instance> key | grep '"username"' # The Redis destination password (`<a9s-redis-destination-password>`)