Forking and Migration
This document describes how to migrate data from an older a9s Redis®[^*] version to an a9s KeyValue. We present two approaches:
- the first one uses the Disaster Recovery feature to create a fork of an existing Service Instance,
- the second one uses different commands via the Valkey CLI to migrate the data.
The following migration paths have been tested and are supported by the a9s Data Services Framework:
- a9s Redis® 7 -> a9s KeyValue 8
Migrate to a9s KeyValue Using the Forking Feature
Follow the steps described in Fork a Service Instance to create a fork of an existing a9s Redis® Service Instance.
Migrate to a9s KeyValue Manually
In the following we present two approaches to migrate to a9s KeyValue manually:
- the first one uses the MIGRATE command,
- while the other one performs an element-wise DUMP & RESTORE.
Prerequisites
- Install valkey-cli.
Known Issues
- Downtime during the migration. It will depend on the amount of data inside the a9s Service Instance and network connection throughput between both a9s Service Instances.
- Every scenario in this migration was tested with a9s Redis®/KeyValue instances following best practices for key naming. So, it is known that there may be problems when trying to migrate that when key names do not follow best practices. An example of this limitation is the use of keys with names containing binary content or other special characters which can potentially cause issues during the migration. Please note that it might be necessary to perform individual migration for any existing special keys, although we do provide an alternative Ruby-based solution in Migrate Using Manual DUMP & RESTORE for such cases.
- If a a9s Redis®/KeyValue Service Instance uses multiple logical database, in order to migrate the whole Redis®/Valkey content, it is necessary to migrate the Redis®/Valkey logical database one after another. A piece of information about logical databases:
Out of the box, a Redis®/Valkey supports 16 logical databases. These databases are effectively siloed off from one another, and when a command is executed in one database, it doesn't affect any of the data stored in other databases in the a9s Redis®/KeyValue Service Instance.
Before Migrating
There are some preliminary measures required before the migration can take place:
- Isolating The Nodes Before The Migration
- Create a New a9s KeyValue Destination Instance
- Create an SSH tunnel
Isolating The Nodes Before The Migration
It is highly recommended to isolate the a9s Redis®/KeyValue Service Instance before starting the migration. This avoids potential data inconsistencies that could be caused if the a9s Redis®/KeyValue Service Instance is receiving new data while processing the migration at the same time.
This can be achieved by unbinding all applications from the a9s Redis®/KeyValue Service Instance.
We recommend saving the Service Instance's RDB file as failsafe in the event of unexpected data loss. This file must be stored safely, at least until the consistency of the database has been confirmed, post-migration.
The latest backup can be downloaded from a9s Service Dashboard.
Details
Save the RDB File (Alternative)
Alternatively, it is possible to download the RDB file for a specific Service Instance usingvalkey-cli:# set `tls` value if the Origin Service Instance is SSL
# origin_tls="--tls --cacert <origin_ca_cert_path>"
valkey-cli --user <a9s-instance-origin-username> -h <a9s-instance-origin-host> -p 6379 \
-a <a9s-instance-origin-password> ${origin_tls:-} --rdb /tmp/dump.rdb
Please refer to the Placeholder Values table at the end of this guide for further details on
<a9s-instance-origin-host>, <a9s-instance-origin-username>, <a9s-instance-origin-password>,
<origin_ca_cert_path> and similar placeholders on this document.
Create a New a9s KeyValue Destination Instance
Create a new and empty a9s KeyValue destination Service Instance that will be the target for the data migration from an existing a9s Redis®/KeyValue origin Service Instance.
- Do not insert any data inside of the machine before the migration and checking are done.
- Create a Service Instance with the same capabilities or greater.
Create an SSH Tunnel
Create a reverse tunnel
The goal is to give access to the Application Developer to the a9s KeyValue Service Instance. And for this, it is necessary to make a SSH tunnels to the a9s Redis®/KeyValue origin Service Instance.
In the end, we want to achieve the following scenario below.
(port 6379) (port 6379)
/---> * Origin Instance * ----- MIGRATE -----> * Destination Instance *
/
* CF Application * (via SSH Tunnel) [infrastructure network]
--------/-----------------------------------------------------------------
/ [Developer network]
* Developer Machine * (local port 6379)
It is possible to access any of the a9s Data Services locally. This means that a local client can connect to the Service Instance for any purpose such as debugging. Cloud Foundry (CF) provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the Cloud Foundry documentation.
First of all an application must be bound to the Service Instance. For more information, see Bind an application to a Service Instance.
cf ssh support must be enabled in the platform. Ask the Platform Operator for further details.
Follow the section Create a Tunnel to The Service to create the reverse tunnel.
Migrate to a9s KeyValue Using the MIGRATE Option
-
Migrating between an SSL Service Instance and a non-SSL Service Instance, or vice versa, is not supported by a9s Redis®/Valkey Service Instance.
-
When migrating data between SSL Service Instances they MUST have the same CA, or it MUST be possible to verify the certificate of the destination with the CA of the origin Service Instance. Otherwise, it will fail because the TLS connection can not be verified.
-
When migrating from a9s Redis Data Service to a9s KeyValue Data Service, you must use the same CA; otherwise, the a9s Service Instances will not use the same CA, and the MIGRATE command will fail due to a certificate issue.
For all of the cases above, please use the Migrate Using Manual DUMP & RESTORE.
The main strategy to migrate the whole dataset from one a9s Redis®/KeyValue Service Instance to a9s KeyValue service instance is based on the Redis® MIGRATE feature/Valkey MIGRATE feature.
This feature comes in the form of a simple command that can be run through redis-cli/valkey-cli. Using the CLI, we
shall connect to the existing instance, and execute the MIGRATE command. The command accepts the hostname and
credentials for the new instance, which implies that there will be a direct connection between
the two Service Instances during the migration.
The vendor documentation of the feature also says:
The command is atomic and blocks the two instances for the time required to transfer the key, at any given time the key will appear to exist in a given instance or the other instance unless a timeout error occurs.
Hence, there will be a system downtime during the migration. Thus it is suggested to trigger the migration process during an appropriate time window; likely in the moment in the week, the day, and the period with less impact for the client of this specific a9s Service Instance. The Application Developer must decide which timeframe fits these criteria.
Based on internal and isolated tests, migrating 1GB takes about 50 seconds on average. However, it will depend on each environment and its particularities. Be aware that it might take some time, so it is necessary to plan wisely.
Perform the Migration
Use the valkey-cli to retrieve the keys and migrate between the a9s Redis® and Valkey Service Instances. Replace all
placeholder values in the command below and execute it afterwards. These commands may change depending on the version of
Redis®/Valkey, make sure to run the appropriate command for the use case. This command is valid for Redis® 7 GA and
above:
Command
# set `tls` value if the origin Service Instance is ssl
# origin_tls="--tls --cacert <origin_ca_cert_path>"
client_origin_command="valkey-cli -h <a9s-instance-origin-host> -p <a9s-instance-origin-port> \
--user <a9s-instance-origin-username> -a <a9s-instance-origin-password> \
-n <a9s-instance-origin-logical-database-number> ${origin_tls:-}"
${client_origin_command} --raw KEYS '*' | xargs ${client_origin_command} MIGRATE \
<a9s-instance-destination-host> <a9s-instance-destination-port> "" <a9s-instance-origin-logical-database-number> \
<timeout> COPY AUTH2 <a9s-instance-destination-username> <a9s-instance-destination-password> KEYS \
> /tmp/migration-output-checker.txt
The migration-output-checker.txt contains the output of the MIGRATE command, and it is used in a further step to
make sure the migration was executed with success. For more information see
Check The Database Consistency.
The command above can only be executed by an Application Developer once they have access to the a9s origin service instance via the reverse tunnel.
Furthermore, this command is composed of multiple commands, all of which serve a specific function:
- The first command,
valkey-cli --raw KEYS '*', lists all the keys stored in the existing instance. - The second command,
xargs valkey-cli MIGRATE <destination-host> <destination-port> "" 0 5000 COPY AUTH <pass> KEYS, uses the output of the first command toMIGRATEthe data of the source host to the destination host.- The part following the first command and preceding the second one simply utilizes the tunnel to connect to the existing instance, where the second command is executed.
- The
COPYargument must be present in the command to avoid deleting the key from the source host.
Please refer to the Placeholder Values table at the end of this guide for more specific details on the placeholders.
When using logical databases this command must be executed as many times as the amount of the logical databases.
It is necessary to change the <a9s-instance-original-logical-database-number> value accordingly.
If the origin a9s Redis®/KeyValue instance contains keys with names containing special characters this migration process will not work. For example, if one key contains a breakline character this can be a problem.