Administrator’s guide

Deploying the cluster

To deploy the cluster, first, configure your Tarantool instances according to the desired cluster topology, for example:

my_app.router: {"advertise_uri": "localhost:3301", "http_port": 8080, "workdir": "./tmp/router"}
my_app.storage_A_master: {"advertise_uri": "localhost:3302", "http_enabled": False, "workdir": "./tmp/storage-a-master"}
my_app.storage_A_replica: {"advertise_uri": "localhost:3303", "http_enabled": False, "workdir": "./tmp/storage-a-replica"}
my_app.storage_B_master: {"advertise_uri": "localhost:3304", "http_enabled": False, "workdir": "./tmp/storage-b-master"}
my_app.storage_B_replica: {"advertise_uri": "localhost:3305", "http_enabled": False, "workdir": "./tmp/storage-b-replica"}

Then start the instances, for example using cartridge CLI:

$ cartridge start my_app --cfg demo.yml --run_dir ./tmp/run --foreground

And bootstrap the cluster. You can do this via the Web interface which is available at http://<instance_hostname>:<instance_http_port> (in this example, http://localhost:8080).

In the web interface, do the following:

1. Depending on the authentication state:

   • If enabled (in production), enter your credentials and click Login:

     

      

   • If disabled (for easier testing), simply proceed to configuring the cluster.

2. Click Сonfigure next to the first unconfigured server to create the first replica set – solely for the router (intended for compute-intensive workloads).

   

    

   In the pop-up window, check the vshard-router role or any custom role that has vshard-router as a dependent role (in this example, this is a custom role named app.roles.api).

   (Optional) Specify a display name for the replica set, for example router.

   

    

   Note

   As described in the built-in roles section, it is a good practice to enable workload-specific cluster roles on instances running on physical servers with workload-specific hardware.

   Click Create replica set and see the newly-created replica set in the web interface:

   

    

   Warning

   Be careful: after an instance joins a replica set, you CAN NOT revert this or make the instance join any other replica set.

3. Create another replica set for a master storage node (intended for transaction-intensive workloads).

   Check the vshard-storage role or any custom role that has vshard-storage as a dependent role (in this example, this is a custom role named app.roles.storage).

   (Optional) Check a specific group, for example hot. Replica sets with vshard-storage roles can belong to different groups. In our example, these are hot or cold groups meant to process hot and cold data independently. These groups are specified in the cluster’s configuration file; by default, a cluster has no groups.

   (Optional) Specify a display name for the replica set, for example hot-storage.

   Click Create replica set.

   

    

4. (Optional) If required by topology, populate the second replica set with more storage nodes:

   1. Click Configure next to another unconfigured server dedicated for transaction-intensive workloads.

   2. Click Join Replica Set tab.

   3. Select the second replica set, and click Join replica set to add the server to it.

      

       

5. Depending on cluster topology:

   • add more instances to the first or second replica sets, or
   • create more replica sets and populate them with instances meant to handle a specific type of workload (compute or transactions).

   For example:

   

    

6. (Optional) By default, all new vshard-storage replica sets get a weight of 1 before the vshard bootstrap in the next step.

Note

In case you add a new replica set after vshard bootstrap, as described in the topology change section, it will get a weight of 0 by default.

To make different replica sets store different numbers of buckets, click Edit next to a replica set, change its default weight, and click Save:

 

For more information on buckets and replica set’s weights, see the vshard module documentation.

1. Bootstrap vshard by clicking the corresponding button, or by saying cartridge.admin.boostrap_vshard() over the administrative console.

   This command creates virtual buckets and distributes them among storages.

   From now on, all cluster configuration can be done via the web interface.

Updating the configuration

Cluster configuration is specified in a YAML configuration file. This file includes cluster topology and role descriptions.

All instances in Tarantool cluster have the same configuration. To this end, every instance stores a copy of the configuration file, and the cluster keeps these copies in sync: as you submit updated configuration in the Web interface, the cluster validates it (and rejects inappropriate changes) and distributes automatically across the cluster.

To update the configuration:

1. Click Configuration files tab.

2. (Optional) Click Downloaded to get hold of the current configuration file.

3. Update the configuration file.

   You can add/change/remove any sections except system ones: topology, vshard, and vshard_groups.

   To remove a section, simply remove it from the configuration file.

4. Compress the configuration file as a .zip archive and click Upload configuration button to upload it.

   You will see a message in the lower part of the screen saying whether configuration was uploaded successfully, and an error description if the new configuration was not applied.

Managing the cluster

This chapter explains how to:

• change the cluster topology,
• enable automatic failover,
• switch the replica set’s master manually,
• deactivate replica sets, and
• expel instances.

tarantool> vshard.storage.info().bucket
---
- receiving: 0
  active: 1000
  total: 1000
  garbage: 0
  sending: 0
...

Expelling instances

Once an instance is expelled, it can never participate in the cluster again as every instance will reject it.

To expel an instance, click … next to it, then click Expel server and Expel:

 

Note

There are two restrictions:

• You can’t expel a leader if it has a replica. Switch leadership first.
• You can’t expel a vshard-storage if it has buckets. Set the weight to zero and wait until rebalancing is completed.

Enabling automatic failover

In a master-replica cluster configuration with automatic failover enabled, if the user-specified master of any replica set fails, the cluster automatically chooses the next replica from the priority list and grants it the active master role (read/write). When the failed master comes back online, its role is restored and the active master, again, becomes a replica (read-only). This works for any roles.

To set the priority in a replica set:

1. Click Edit next to the replica set in question.

2. Scroll to the bottom of the Edit replica set box to see the list of servers.

3. Drag replicas to their place in the priority list, and click Save:

   

    

The failover is disabled by default. To enable it:

1. Click Failover:

   

    

2. In the Failover control box, click Enable:

   

    

The failover status will change to enabled:

 

For more information, see the replication section of the Tarantool manual.

Managing users

On the Users tab, you can enable/disable authentication as well as add, remove, edit, and view existing users who can access the web interface.

Notice that the Users tab is available only if authorization in the web interface is implemented.

Also, some features (like deleting users) can be disabled in the cluster configuration; this is regulated by the auth_backend_name option passed to cartridge.cfg().

Resolving conflicts

Tarantool has an embedded mechanism for asynchronous replication. As a consequence, records are distributed among the replicas with a delay, so conflicts can arise.

To prevent conflicts, the special trigger space.before_replace is used. It is executed every time before making changes to the table for which it was configured. The trigger function is implemented in the Lua programming language. This function takes the original and new values of the tuple to be modified as its arguments. The returned value of the function is used to change the result of the operation: this will be the new value of the modified tuple.

For insert operations, the old value is absent, so nil is passed as the first argument.

For delete operations, the new value is absent, so nil is passed as the second argument. The trigger function can also return nil, thus turning this operation into delete.

This example shows how to use the space.before_replace trigger to prevent replication conflicts. Suppose we have a box.space.test table that is modified in multiple replicas at the same time. We store one payload field in this table. To ensure consistency, we also store the last modification time in each tuple of this table and set the space.before_replace trigger, which gives preference to newer tuples. Below is the code in Lua:

fiber = require('fiber')
-- define a function that will modify the function test_replace(tuple)
        -- add a timestamp to each tuple in the space
        tuple = box.tuple.new(tuple):update{{'!', 2, fiber.time()}}
        box.space.test:replace(tuple)
end
box.cfg{ } -- restore from the local directory
-- set the trigger to avoid conflicts
box.space.test:before_replace(function(old, new)
        if old ~= nil and new ~= nil and new[2] < old[2] then
                return old -- ignore the request
        end
        -- otherwise apply as is
end)
box.cfg{ replication = {...} } -- subscribe

Monitoring a cluster via CLI

This section describes parameters you can monitor over the administrative console.

$ tarantoolctl connect <instance_hostname>:<port>

tarantool> vshard.storage.info()
---
- replicasets:
    <replicaset_2>:
    uuid: <replicaset_2>
    master:
        uri: storage:storage@127.0.0.1:3303
    <replicaset_1>:
    uuid: <replicaset_1>
    master:
        uri: storage:storage@127.0.0.1:3301
  bucket: <!-- buckets status
    receiving: 0 <!-- buckets in the RECEIVING state
    active: 2 <!-- buckets in the ACTIVE state
    garbage: 0 <!-- buckets in the GARBAGE state (are to be deleted)
    total: 2 <!-- total number of buckets
    sending: 0 <!-- buckets in the SENDING state
  status: 1 <!-- the status of the replica set
  replication:
    status: disconnected <!-- the status of the replication
    idle: <idle>
  alerts:
  - ['MASTER_IS_UNREACHABLE', 'Master is unreachable: disconnected']

tarantool> vshard.router.info()
---
- replicasets:
    <replica set UUID>:
      master:
        status: <available / unreachable / missing>
        uri: <!-- URI of master
        uuid: <!-- UUID of instance
      replica:
        status: <available / unreachable / missing>
        uri: <!-- URI of replica used for slave requests
        uuid: <!-- UUID of instance
      uuid: <!-- UUID of replica set
    <replica set UUID>: ...
    ...
  status: <!-- status of router
  bucket:
    known: <!-- number of buckets with the known destination
    unknown: <!-- number of other buckets
  alerts: [<alert code>, <alert description>], ...

Upgrading schema

When upgrading Tarantool to a newer version, please don’t forget to:

1. Stop the cluster
2. Make sure that upgrade_schema option is enabled
3. Start the cluster again

This will automatically apply box.schema.upgrade() on the leader, according to the failover priority in the topology configuration.

Disaster recovery

Please see the disaster recovery section in the Tarantool manual.

Backups

Please see the backups section in the Tarantool manual.