Live upgrade from Tarantool 1.6 to 1.10
This page includes explanations and solutions to some common issues when upgrading a replica set from Tarantool 1.6 to 1.10.
Versions later that 1.6 have incompatible .snap and .xlog file formats: 1.6 files are supported during upgrade, but you won’t be able to return to 1.6 after running under 1.10 or 2.x for a while. A few configuration parameters are also renamed.
To perform a live upgrade from Tarantool 1.6 to a more recent version, like 2.8.4, 2.10.1 and such, it is necessary to take an intermediate step by upgrading 1.6 -> 1.10 -> 2.x. This is the only way to perform the upgrade without downtime.
However, a direct upgrade of a replica set from 1.6 to 2.x is also possible, but only with downtime.
The procedure of live upgrade from 1.6 to 1.10 is similar to the general cluster upgrade procedure, but with slight differences in the Upgrading storages step. Find below the general storage upgrade procedure and the 1.6-specific notes for its steps.
Upgrade storage instances by performing the following steps for each replica set:
Примечание
To detect possible upgrade issues early, we recommend that you perform a replication check on all instances of the replica set after each step.
- Pick a replica (a read-only instance) from the replica set. Stop this replica
and start it again on the target Tarantool version. Wait until it reaches the
running
status (box.info.status == running
). - Restart all other read-only instances of the replica set on the target version one by one.
- Make one of the updated replicas the new master using the applicable instruction from Switching the master.
- Restart the last instance of the replica set (the former master, now a replica) on the target version.
- Run box.schema.upgrade() on the new master. This will update the Tarantool system spaces to match the currently installed version of Tarantool. The changes will be propagated to other nodes via the replication mechanism later.
- Run
box.snapshot()
on every node in the replica set to make sure that the replicas immediately see the upgraded database state in case of restart.
Replication check: New Tarantool nodes follow 1.6 nodes just fine, but some 1.6 nodes might disconnect from new nodes with an ER_LOADING error. This is not critical, the error goes away when replication on 1.6 is restarted:
old_repl = box.cfg.replication box.cfg{replication = ""} box.cfg{replication = old_repl}
Point of no return: When upgrading from Tarantool 1.6, the step 3 (switching the master) is the point of no return. Оnce you complete it, the schema is no longer compatible with the initial version.
Restarting on the target version (steps 1, 2, and 4): Tarantool 1.10+ fails to recover from 1.6 xlogs, unless
box.cfg{force_recovery = true}
is set. There is a slight difference between 1.6 and 1.10 xlogs, which makes 1.6 xlogs appear erroneous to 1.10+ instances. In order to work around this, start the instance inforce_recovery
mode. To do so, add the lineforce_recovery = true
to the file where the instance is initialized – for example, toinit.lua
.Running box.schema.upgrade() (step 5): There was a breaking change between 1.6 and 1.10 – in 1.6, the field type
num
was an alias tonumber
, and in 1.10,num
is converted tounsigned
. This means that afterbox.schema.upgrade()
is performed on the master, the user might have some spaces withunsigned
fields containing non-unsigned values:double
,int
, and so on. This will make the snapshot inconsistent, unless an extra action is performed afterbox.schema.upgrade()
. Run this code in the Tarantool console on the new master:-- First find all spaces containing unsigned fields with non-unsigned values in them. -- Say, we have one such space denoted problematic_space and the problem is in field problematic_field_no. a = box.space.problematic_space:format() a[problematic_field_no].type = 'number' box.space.problematic_space:format(a)
Taking snapshots (step 6): The user might be concerned with snapshot size in 1.10 – it’s drastically smaller than the one created by 1.6 (for example, ~300 Mb vs. 6 Gb in some corner cases). There is nothing to worry about. Tarantool 1.6 didn’t compress snapshots, while Tarantool 1.10 and above does that.