Migrations

Simple migrations

There are two types of schema migration that do not require data migration:

• Creating an index. A new index can be created at any time. To learn more about index creation, see Indexes and the space_object:create_index() reference.

• Adding a field to the end of a space. To add a field, update the space format so that it includes all its fields and also the new field. For example:

  local users = box.space.writers
  local fmt = users:format()
  
  table.insert(fmt, { name = 'age', type = 'number', is_nullable = true })
  users:format(fmt)
  

  The field must have the is_nullable parameter. Otherwise, an error occurs if the space contains tuples of old format.

  Note

  After creating a new field, you probably want to fill it with data. The tarantool/moonwalker module is useful for this task.

Complex migrations

Other types of migrations are more complex and require additional actions to maintain data consistency.

Migrations are possible in two cases:

• When Tarantool starts, and no client uses the database yet
• During request processing, when active clients are already using the database

For the first case, it is enough to write and test the migration code. The most difficult task is to migrate data when there are active clients. You should keep it in mind when you initially design the data schema.

We identify the following problems if there are active clients:

• Associated data can change atomically.
• The system should be able to transfer data using both the new schema and the old one.
• When data is being transferred to a new space, data access should consider that the data might be in one space or another.
• Write requests must not interfere with the migration. A common approach is to write according to the new data schema.

These issues may or may not be relevant depending on your application and its availability requirements.

Tarantool offers the following features that make migrations easier and safer:

• Transaction mechanism. It is useful when writing a migration, because it allows you to work with the data atomically. But before using the transaction mechanism, you should explore its limitations. For details, see the section about transactions.
• space:upgrade() function (EE only). With the help of space:upgrade(), you can enable compression and migrate, including already created tuples. For details, check the Upgrading space schema section.
• Centralized migration management mechanism (EE only). Implemented in the Enterprise version of the tt utility and in Tarantool Cluster Manager, this mechanism enables migration execution and tracking in the replication clusters. For details, see Centralized migration management.

Applying migrations

The migration code is executed on a running Tarantool instance. Important: no method guarantees you transactional application of migrations on the whole cluster.

Method 1: include migrations in the application code

This is quite simple: when you reload the code, the data is migrated at the right moment, and the database schema is updated. However, this method may not work for everyone. You may not be able to restart Tarantool or update the code using the hot-reload mechanism.

Method 2: the tt utility

Connect to the necessary instance using tt connect.

$ tt connect admin:password@localhost:3301

• If your migration is written in a Lua file, you can execute it using dofile(). Call this function and specify the path to the migration file as the first argument. It looks like this:

  tarantool> dofile('0001-delete-space.lua')
  ---
  ...
  

• (or) Copy the migration script code, paste it into the console, and run it.

You can also connect to the instance and execute the migration script in a single call:

$ tt connect admin:password@localhost:3301 -f 0001-delete-space.lua

Centralized migration management

Tarantool EE offers a mechanism for centralized migration management in replication clusters that use etcd as a configuration storage. The mechanism uses the same etcd storage to store migrations and applies them across the entire Tarantool cluster. This ensures migration consistency in the cluster and enables migration history tracking.

The centralized migration management mechanism is implemented in the Enterprise version of the tt utility and in Tarantool Cluster Manager.

To learn how to manage migrations in Tarantool EE clusters from the command line, see Centralized migrations with tt. To learn how to use the mechanism from the TCM web interface, see the Performing migrations TCM documentation page.