Tarantool Enterprise documentation

Changelog

1.10.3-5 - 2019-04-05

  • The underlying Tarantool Community version has been upgraded to 1.10.3-6-gfbf53b9.
  • The following features have been added:
    • Failover priority configuration via the web interface.
    • RPC module (remote calls between cluster instances).
    • Space explorer via the web interface.
    • OpenLDAP client for Tarantool.
  • Instance restart now triggers configuration validation before the roles initialization.
  • The web interface design has been updated.

1.10.2-41 - 2019-02-08

  • The underlying Tarantool Community version has been upgraded to 1.10.2-131-g6f96bfe.
  • The cluster template has been updated to allow the application to operate across several hosts (virtual machines).
  • The following closed-source modules of latest versions have been added: queue, front, errors, membership, cluster, and oracle. These rocks include:
    • a new front-end core with cosmetic changes;
    • active master indication during failover;
    • the ability to disable the vshard-storage role after the end of the rebalancing process;
    • dependencies updates and minor improvements.

1.10.2-15 - 2018-12-13

  • The underlying Tarantool Community version has been upgraded to 1.10.2-84-g19d471bd4.
  • The checks module has been updated in the bundle.
  • The custom (user-defined) cluster roles API has been added to the cluster module.
  • The vshard replica set’s weight parameter is now supported.

1.10.2-4 — 2018-10-31

  • Sample applications demonstrating how to run Tarantool in Docker and a write-through cache to PostgreSQL have been added.
  • The abilities to manually switch the replica set’s master and enable automatic failover have been added to Web interface.
  • The tarantoolapp utility that helps set up a development environment and pack the application in an environment-independent way has been added.

1.10.1-29 — 2018-10-04

  • The sample application is now cluster-based and the cluster is orchestrated via a Web interface.
  • Tarantool Enterprise release comes in an archive that includes an offline rocks repository from which you can install all necessary modules.
  • Oracle connector is now included in the rocks repository. With this package, Lua applications can access Oracle databases.

1.10.1 — 2018-04-09

  • The Tarantool build is now static: all dependencies are linked into a static binary. This simplifies deploy of Tarantool in the Linux environment.
  • The underlying Tarantool Community version has been upgraded to 1.10.1. For all new features and bugfixes of the Community version please refer to https://github.com/tarantool/tarantool/releases.
  • The modules required for integration with ZooKeeper and orchestrator are now deprecated and no longer supported.

Basics of Enterprise clustering

Tarantool Enterprise allows you to cluster Tarantool instances. An Enterprise cluster is a collection of Tarantool instances acting in concert. While a single Tarantool instance can leverage the performance of a single server and is vulnerable to failure, the cluster spans multiple servers, utilizes their cumulative CPU power, and is fault-tolerant.

To fully utilize the capabilities of the Tarantool Enterprise cluster, one has to develop applications keeping in mind they are to run in a cluster environment. Cluster-aware application development is not a trivial task, so Tarantool Enterprise bundle, as a software development kit (SDK), provides you with utilities and templates to help:

  • easily set up a development environment for your applications;
  • plug the necessary open and closed source Lua modules;
  • pack the applications in an environment-independent way: together with module binaries and Tarantool executables.

The resulting package can be installed and started on one or multiple servers as one or multiple instantiated services, independent or organized into a cluster.

Tarantool Enterprise cluster provides your cluster-aware applications with the following key benefits:

  • horizontal scalability and load balancing via built-in automatic sharding;
  • asynchronous replication;
  • automatic failover;
  • centralized cluster control via GUI or API;
  • automatic configuration synchronization;
  • instance functionality segregation.

Tarantool Enterprise cluster can segregate functionality between instances via built-in and custom (user-defined) cluster roles. They can be toggled on and off for instances on the fly during cluster operation. This allows you to put different types of workloads (e.g., compute- and transaction-intensive ones) on different physical servers with dedicated hardware.

Additionally, Tarantool Enterprise has third-party database connectivity (Oracle, Microsoft SQL Server, IBM DB2) and a built-in audit log to help you harden security.

Setup

This chapter explains how to download and set up Tarantool Enterprise and run a sample application provided with it.

System requirements

The recommended system requirements for running Tarantool Enterprise are as follows.

Hardware requirements

To fully ensure the fault tolerance of a distributed data storage system, at least three physical computers or virtual servers are required. For testing/development purposes, the system can be deployed using a smaller number of servers; however, it is not recommended to use such configurations for production.

Software requirements

Tarantool Enterprise supports Red Hat Enterprise Linux 7 and CentOS 7 as host operating systems.

Note

Tarantool Enterprise can run on other systemd-based Linux distributions but it is not tested on them and may not work as expected.

Python and Python connector to Tarantool are required to launch out-of-the-box tests provided for the sample application. To install them, e.g., on CentOS, say:

$ yum -y install epel-release
$ yum -y install python-pip
$ pip install tarantool requests

Network requirements

Hereinafter, “storage servers” or “Tarantool servers” are the computers used to store and process data, and “administration server” is the computer used by the system operator to install and configure the product.

The Tarantool cluster has a full mesh topology, therefore all Tarantool servers should be able to communicate and send traffic from any TCP port to TCP ports 3000:4000.

To configure remote monitoring or to connect via the administrative console, the administration server should be able to access the following TCP ports on Tarantool servers:

  • 22 to use the SSH protocol,
  • 8080 to monitor the HTTP-metrics.

Additional network configuration recommendations

It is recommended to apply the following settings for sysctl on all Tarantool servers:

$ # TCP KeepAlive setting
$ sysctl -w net.ipv4.tcp_keepalive_time=60
$ sysctl -w net.ipv4.tcp_keepalive_intvl=5
$ sysctl -w net.ipv4.tcp_keepalive_probes=5

This optional setup of the Linux network stack helps speed up the troubleshooting of network connectivity when the server physically fails. To achieve the maximum performance, you may also need to configure other network stack parameters that are not specific to the Tarantool DBMS. For more information, please refer to the Network Performance Tuning Guide section of the RHEL7 user documentation.

Package contents

The latest release package of Tarantool Enterprise is available in the S3 repository. Please contact support@tarantool.org for the details.

The package is distributed as a tar + gzip archive and includes the following components and features:

  • static Tarantool binary for simplified deployment in Linux environments,
  • built-in audit log,
  • selection of open and closed source modules,
  • sample application walking you through all included modules.

Archive contents:

  • tarantool is the main executable of Tarantool.
  • tarantoolctl is the utility script for installing supplementary modules.
  • tarantoolapp is the utility script to help you set up a development environment for applications and pack them for easy deployment.
  • example/ is the directory containing sample applications:
    • cluster/ is an application designed to demonstrate key features and modules included in the distribution;
    • pg_writethrough_cache/ is an application showcasing how Tarantool can cache data written to, for example, a PostgreSQL database;
    • ora_writebehind_cache/ is an application showcasing how Tarantool can cache writes and queue them to, for example, an Oracle database;
    • docker/ is an application designed to be easily packed into a Docker container;
  • rocks/ is the directory containing a selection of additional open and closed source modules included in the distribution as an offline rocks repository. See the rocks reference for details.
  • templates/ is the directory containing template files for your application development environment.
  • deprecated/ is a set of modules that are no longer actively supported:
    • vshard-zookeeper-orchestrator is a Python application for launching orchestrator,
    • zookeeper-scm files are the ZooKeeper integration modules (require usr/ libraries).

Unpacking the archive

The delivered archive should be uploaded to a server and unpacked:

$ tar xvf tarantool-enterprise-bundle-<version>.tar.gz

No further installation is required as the unpacked binaries are almost ready to go. Go to the directory with the binaries and add them to the executable path by running the script provided by the distribution:

$ source ./env.sh

Developer’s guide

This guide explains how to develop environment-independent applications – either cluster-aware or not – and run sample applications provided by the distribution.

An environment-independent application is an assembly (in one directory) of:

  • files with Lua code,
  • tarantool executable,
  • plugged external modules (if necessary).

When started by the tarantool executable, the application provides a service.

The modules are Lua rocks installed into a virtual environment (under the application directory) similar to Python’s virtualenv and Ruby’s bundler.

Such an application has the same structure both in development and production-ready phases. All the application-related code resides in one place, ready to be packed and copied over to any server.

To develop and run an application, in short, you need to go through the following steps:

  1. Set up a development environment from a template for the application.
  2. Develop the application. In case it is a cluster-aware application, implement it in a custom cluster role to initialize the database in a cluster environment.
  3. Plug all the necessary rock modules.
  4. Pack the application and module binaries together with the tarantool executable.
  5. Upload, install, and start corresponding instantiated services on every server dedicated for Tarantool Enterprise.
  6. In case it is a cluster-aware application, deploy the cluster.

The following sections provide details for each of these steps.

Setting up development environments from templates

Tarantool Enterprise provides you with templates to help set up your application development environment for both cluster-aware and plain (e.g., Tarantool as a proxy to third-party databases) application use cases.

To set up a development environment, say:

$ tarantoolapp create --template [plain|cluster] --name <app_name>

where specify:

  • the plain template to develop an application for single or multiple independent Tarantool instances; or
  • the cluster template to develop a cluster-aware application.

The script will automatically set up a Git repository in a new <app_name>/ directory, tag it with version 0.1.0, and put the necessary files into it.

In this Git repository, you can develop the application, plug the necessary modules, and then easily pack everything to deploy on your server(s).

Developing applications

This section describes templates, default files they provide, and introduces the notion of cluster roles that allow you to segregate functionality between instances.

To start developing an application, simply edit the default files provided by the template selected in the previous step.

During development, keep track of the application version.

Plain template

The plain template creates the <app_name>/ directory with the following contents:

  • <app_name>-scm-1.rockspec file where you can specify the application dependencies.
  • deps.sh script that resolves dependencies from the .rockspec file.
  • init.lua file which is the entry point for your application.
  • .git file necessary for a Git repository.
  • .gitignore file to ignore the unnecessary files.

Cluster template

In addition to the files listed in the plain template section, the cluster template contains the following:

  • env.lua file that sets common rock paths so that the application can be started from any directory.
  • custom-role.lua file that is a placeholder for a user-defined cluster role described in the next section.

The entry point file (init.lua) of the cluster template differs from the plain one. Among other things, it loads the cluster module and calls its initialization function:

...
local cluster = require('cluster')
...
cluster.cfg({
  workdir = ...,
  advertise_uri = ...,
  ...
})
...

The cluster.cfg() call renders the instance operable via the administrative console but does not call box.cfg() to configure instances.

Warning

Calling the box.cfg() function is forbidden.

The cluster itself will do it for you when it is time to:

  • bootstrap the current instance once you:
    • run cluster.bootstrap() via the administrative console, or
    • click Create in the web interface;
  • join the instance to an existing cluster once you:
    • run cluster.join_server({uri = 'other_instance_uri'}) via the console, or
    • click Join (an existing replica set) or Create (a new replica set) in the web interface.

Before developing a cluster-aware application, familiarize yourself with the notion of cluster roles described in the next section and make sure to define a custom role to initialize the database for the cluster application.

Defining custom cluster roles

Tarantool Enterprise cluster segregates instance functionality in a role-based way. Cluster roles are Lua modules that implement some instance-specific functions and/or logic.

Since all instances running cluster applications use the same source code and are aware of all the defined roles (and plugged modules), multiple different roles can be dynamically enabled and disabled on any number of instances without restarts even during cluster operation.

Built-in roles

The cluster module comes with two built-in roles that implement automatic sharding:

  • vshard-router that handles the vshard’s compute-intensive workload: routes requests to storage nodes.

  • vshard-storage that handles the vshard’s transaction-intensive workload: stores and manages a subset of a dataset.

    Note

    For more information on sharding, see the vshard module documentation.

With the built-in and custom roles, Tarantool Enterprise allows you to develop applications with separated compute and transaction handling. Later, the relevant workload-specific roles can be enabled on different instances running on physical servers with workload-dedicated hardware.

Neither vshard-router nor vshard-storage manage spaces, indexes, or formats. To start developing an application, edit the custom-role.lua placeholder file: add a box.schema.space.create() call to your first cluster role.

Additionally, you can implement several such roles to:

  • define stored procedures;
  • implement functionality on top of vshard;
  • go without vshard at all;
  • implement one or multiple supplementary services such as e-mail notifier, replicator, etc.

Implementing and registering custom roles

To implement a custom cluster role, do the following:

  1. Register the new role in the cluster by modifying the cluster.cfg() call in the init.lua entry point file:

    ...
    local cluster = require('cluster')
    ...
    cluster.cfg({
      workdir = ...,
      advertise_uri = ...,
      roles = {'custom-role'},
    })
    ...
    

    where custom-role is the name of the Lua module to be loaded.

  2. Implement the role in a file with the appropriate name (custom-role.lua). For example:

    #!/usr/bin/env tarantool
    -- Custom role implementation
    local role_name = 'custom-role'
    
    local function init()
    ...
    end
    
    local function stop()
    ...
    end
    
    return {
        role_name = role_name,
        init = init,
        stop = stop,
    }
    

    Where the role_name may differ from the module name passed to the cluster.cfg() function. If the role_name variable is not specified, the module name is the default value.

    Note

    Role names must be unique as it is impossible to register multiple roles with the same name.

The role module does not have required functions but the cluster may execute the following ones during the role’s life cycle:

  • init() is the role’s initialization function.

    Inside the function’s body you can call any box functions: create spaces, indexes, grant permissions, etc. Here is what the initialization function may look like:

    local function init(opts)
        -- The cluster passes an 'opts' Lua table containing an 'is_master' flag.
        if opts.is_master then
            local customer = box.schema.space.create('customer',
                { if_not_exists = true }
            )
            customer:format({
                {'customer_id', 'unsigned'},
                {'bucket_id', 'unsigned'},
                {'name', 'string'},
            })
            customer:create_index('customer_id', {
                parts = {'customer_id'},
                if_not_exists = true,
            })
        end
    end
    

    Note

    The function’s body is wrapped in a conditional statement that lets you call box functions on masters only. This protects against replication collisions as data propagates to replicas automatically.

  • stop() is the role’s termination function. Implement it if initialization starts a fiber that has to be stopped or does any job that has to be undone on termination.

  • validate_config() and apply_config() are validation and application functions that make custom roles configurable. Implement them if some configuration data has to be stored cluster-wide.

Next, get a grip on the role’s life cycle to implement the necessary functions.

Role’s life cycle and the order of function execution

The cluster displays all custom role names along with the built-in vshard ones in the web interface. Cluster administrators can enable and disable them for particular instances either via the web interface or cluster public API. For example:

cluster.admin.edit_replicaset('replicaset-uuid', {roles = {'vshard-router', 'custom-role'}})

If multiple roles are enabled on an instance at the same time, the cluster first initializes the built-in roles (if any) and then the custom ones (if any) in the order the latter were listed in cluster.cfg().

The cluster calls the role’s functions in the following circumstances:

  • The init() function, typically, once: either when the role is enabled by the administrator or at the instance restart. Enabling a role once is normally enough.
  • The stop() function – only when the administrator disables the role, not on instance termination.
  • The validate_config() function, first, before the automatic box.cfg() call (database initialization), then – upon every configuration update.
  • The apply_config() function upon every configuration update.

Hence, if the cluster is tasked with performing the following actions, it will execute the functions listed in the following order:

  • Join an instance or create a replica set, both with an enabled role:
    1. validate_config()
    2. init()
    3. apply_config()
  • Restart an instance with an enabled role:
    1. validate_config()
    2. init()
    3. apply_config()
  • Disable role: stop().
  • Upon the cluster.confapplier.patch_clusterwide() call:
    1. validate_config()
    2. apply_config()
  • Upon a triggered failover:
    1. validate_config()
    2. apply_config()

Considering the described behavior:

  • The init() function may:
    • Call box functions.
    • Start a fiber and, in this case, the stop() function should take care of the fiber’s termination.
    • Configure the built-in HTTP server.
    • Execute any code related to the role’s initialization.
  • The stop() finctions must undo any job that has to be undone on role’s termination.
  • The validate_config() function must validate any configuration change.
  • The apply_config() function may execute any code related to a configuration change, e.g., take care of an expirationd fiber.

The validation and application functions together allow you to customize the cluster-wide configuration as described in the next section.

Configuring custom roles

Every instance in the cluster stores a copy of the configuration file in its working directory (configured by cluster.cfg({workdir = ...})):

  • /var/lib/tarantool/<instance_name>/config.yml for instances deployed from RPM packages and managed by systemd.
  • /home/<username>/tarantool_state/var/lib/tarantool/config.yml for instances deployed from archives and managed by tarantoolctl.

The cluster’s configuration is a Lua table. If some application-specific configuration data, e.g., a database schema as defined by DDL (data definition language), has to be stored on every instance in the cluster, you can implement your own API by adding a custom section to the table. The cluster will help you spread it safely across all instances.

Such section goes in parallel (in the same file) with the topology-specific and vshard-specific ones the cluster automatically generates. Unlike the generated, the custom section’s modification, validation, and application logic has to be defined.

The common way is to:

  1. Implement some setters (and getters, if necessary) using the cluster.confapplier public API functions: get_readonly(section), get_deepcopy(section), and patch_clusterwide({section = section_cfg}).

  2. Define, first, the validate_config(conf_new, conf_old), then the apply_config(conf, opts) functions.

    These functions both take two Lua tables as arguments: the ones dubbed with conf are configuration tables as you might have guessed, and opts includes a boolean is_master flag described later.

    Important

    The validate_config() function must detect all configuration problems that may lead to apply_config() errors. For more information, see the next section.

When implementing validation and application functions that call box ones for some reason, the following precautions apply:

  • Due to the role’s life cycle, the cluster does not guarantee an automatic box.cfg() call prior to calling validate_config().

    If the validation function is to call any box functions (e.g., to check a format), make sure the calls are wrapped in a protective conditional statement that checks if box.cfg() has already happened:

    -- Inside the validation function:
    
    if type(box.cfg) == 'function' then
    
        -- Here you can call box functions
    
    end
    
  • Unlike the validation and similar to initialization function, apply_config() can call box functions freely as the cluster applies custom configuration after the automatic box.cfg() call.

    However, creating spaces, users, etc., can cause replication collisions when performed on both master and replica instances simultaneously. The appropriate way is to call such box functions on masters only and let the changes propagate to replicas automatically.

    Upon the apply_config(conf, opts) execution, the cluster passes an is_master flag in the opts table which you can use to wrap collision-inducing box functions in a protective conditional statement:

    -- Inside the configuration application function:
    
    if opts.is_master then
    
        -- Here you can call box functions
    
    end
    

Custom configuration example

Consider the following code as part of the role’s module (custom-role.lua) implementation:

#!/usr/bin/env tarantool
-- Custom role implementation

local cluster = require('cluster')

local role_name = 'custom-role'

-- Modify the config by implementing some setter
local function set_secret(secret)
    local custom_role_cfg = cluster.confapplier.get_deepcopy(role_name) or {}
    custom_role_cfg.secret = secret
    cluster.confapplier.patch_clusterwide({
        [role_name] = custom_role_cfg,
    })
end
-- Validate
local function validate_config(cfg)
    local custom_role_cfg = cfg[role_name] or {}
    if custom_role_cfg.secret ~= nil then
        assert(type(custom_role_cfg.secret) == 'string', 'custom-role.secret must be a string')
    end
    return true
end
-- Apply
local function apply_config(cfg)
    local custom_role_cfg = cfg[role_name] or {}
    local secret = custom_role_cfg.secret or 'default-secret'
    -- Make use of it
end

return {
    role_name = role_name,
    set_secret = set_secret,
    validate_config = validate_config,
    apply_config = apply_config,
}

Once the configuration is customized, do one of the following:

Applying custom role’s configuration

With the implementation showed by the example, you can call the set_secret() function to apply the new configuration via the administrative console or an HTTP endpoint if the role exports one.

The set_secret() function calls cluster.confapplier.patch_clusterwide() which performs a two-phase commit:

  1. It patches the active configuration in memory: copies the table and replaces the "custom-role" section in the copy with the one given by the set_secret() function.
  2. The cluster checks if the new configuration can be applied on all instances except disabled and expelled. All instances subject to update must be healthy and alive according to the membership module.
  3. (Preparation phase) The cluster propagates the patched configuration. Every instance validates it with the validate_config() function of every registered role. Depending on the validation’s result:
    • If successful (i.e., returns true), the instance saves the new configuration to a temporary file named config.prepare.yml within the working directory.
    • (Abort phase) Otherwise, the instance reports an error and all other instances roll back the update: remove the file they may have already prepared.
  4. (Commit phase) Upon successful preparation of all instances, the cluster commits the changes. Every instance:
    1. Creates the active configuration’s hard-link.
    2. Atomically replaces the active one with the prepared. The atomic replacement is indivisible – it can either succeed or fail entirely, never partially.
    3. Calls the apply_config() function of every registered role.

If any of these steps fail, an error pops up in the web interface next to the corresponding instance. The cluster does not handle such errors automatically, they require manual repair.

You will avoid the repair if the validate_config() function can detect all configuration problems that may lead to apply_config() errors.

Using the built-in HTTP server

The cluster launches an httpd server instance during initialization (cluster.cfg()). You can bind a port to the instance via an environmental variable:

-- Get the port from an environmental variable or the default one:
local http_port = os.getenv('HTTP_PORT') or '8080'

local ok, err = cluster.cfg({
   ...
   -- Pass the port to the cluster:
   http_port = http_port,
   ...
})

To make use of the httpd instance, access it and configure routes inside the init() function of some role, e.g. a role that exposes API over HTTP:

local function init(opts)

...

   -- Get the httpd instance:
   local httpd = cluster.service_get('httpd')
   if httpd ~= nil then
       -- Configure a route to, for example, metrics:
       httpd:route({
               method = 'GET',
               path = '/metrics',
               public = true,
           },
           function(req)
               return req:render({json = stat.stat()})
           end
       )
   end
end

For more information on the usage of Tarantool’s HTTP server, see its documentation.

Implementing authorization in the web interface

To implement authorization in the web interface of every instance in Tarantool cluster:

  1. Implement a new, say, auth module with a check_password function. It should check the credentials of any user trying to log in to the web interface.

    The check_password function accepts a username and password and returns an authentication success or failure.

    -- auth.lua
    
    -- Add a function to check the credentials
    local function check_password(username, password)
    
        -- Check the credentials any way you like
    
        -- Return an authentication success or failure
        if not ok then
            return false
        end
        return true
    end
    ...
    

    If you run an LDAP server in your organization, you can connect Tarantool Enterprise to it and let it handle the authorization. In this case add the ldap module to the .rockspec file as a dependency and consider implementing the check_password function the following way:

    -- auth.lua
    
    -- Require the LDAP module at the start of the file
    local ldap = require('ldap')
    ...
    -- Add a function to check the credentials
    local function check_password(username, password)
    
        -- Configure the necessary LDAP parameters
        local user = string.format("cn=%s,ou=superheros,dc=glauth,dc=com", username)
    
        -- Connect to the LDAP server
        local ld, err = ldap.open("localhost:3893", user, password)
    
        -- Return an authentication success or failure
        if not ld then
            return false
        end
        return true
    end
    ...
    
  2. Pass the implemented auth module name as a parameter to cluster.cfg(), so the cluster can use it:

    -- init.lua
    
    local ok, err = cluster.cfg({
        auth_backend_name = 'auth',
        -- The cluster will automatically call 'require()' on the 'auth' module.
        ...
    })
    

    This adds a Log in button to the upper right corner of the web interface but still lets the unsigned users interact with the interface. This is convenient for testing.

    Note

    Also, to authorize requests to cluster API, you can use the HTTP basic authorization header.

  3. To require the authorization of every user in the web interface even before the cluster bootstrap, add the following line:

    -- init.lua
    
    local ok, err = cluster.cfg({
        auth_backend_name = 'auth',
        auth_enabled = true,
        ...
    })
    

    With the authentication enabled and the auth module implemented, the user will not be able to even bootstrap the cluster without logging in. After the successful login and bootstrap, the authentication can be enabled and disabled cluster-wide in the web interface and the auth_enabled parameter is ignored.

Application versioning

Tarantool Enterprise understands semantic versioning as described at semver.org. When developing an application, create new Git branches and tag them appropriately. These tags are used to calculate version increments for subsequent packaging.

For example, if your application has version 1.2.1, tag your current branch with 1.2.1 (annotated or not).

To retrieve the current version from Git, say:

$ git describe --long --tags
1.2.1-12-g74864f2

This output shows that we are 12 commits after the version 1.2.1. If we are to package the application at this point, it will have a full version of 1.2.1-12 and its package will be named <app_name>-1.2.1-12.rpm.

Non-semantic tags are prohibited. You will not be able to create a package from a branch with the latest tag being non-semantic.

Once the application is developed, pack it as described below.

Packaging applications

Once custom cluster role(s) are defined and the application is developed, pack it and all its dependencies (module binaries) together with the tarantool executable.

This will allow you to upload, install, and run your application on any server in one go.

To pack the application, say:

$ tarantoolapp pack [rpm|tgz] <app_name>

where specify:

  • (Recommended) rpm to create an RPM package.
  • tgz to create a tar + gz archive. Choose this option only if you do not have root privileges on servers dedicated for Tarantool Enterprise.

This will create a package (or compressed archive) named <app_name>-<version_tag>-<number_of_commits> (e.g., myapp-1.2.1-12.rpm) containing your environment-independent application.

Proceed to deploying packaged applications (or archived ones) on your servers.

Deploying packaged applications

To deploy your packaged application, do the following on every server dedicated for Tarantool Enterprise:

  1. Upload the package created in the previous step.

  2. Install:

    $ yum install <app_name>-<version>.rpm
    
  3. Start one or multiple Tarantool instances with the corresponding services as described below.

    • A single instance:

      $ systemctl start <app_name>
      

      This will start an instantiated systemd service that will listen to port 3301.

    • Multiple instances on one or multiple servers:

      $ systemctl start <app_name>@instance_1
      $ systemctl start <app_name>@instance_2
      ...
      $ systemctl start <app_name>@instance_<number>
      

      where <app_name>@instance_<number> is the instantiated service name for systemd with an incremental <number> (unique for every instance) to be added to the 3300 port the instance will listen to (e.g., 3301, 3302, etc.).

  4. In case it is a cluster-aware application, proceed to deploying the cluster.

To stop all services on a server, use the systemctl stop command and specify instance names one by one. For example:

$ systemctl stop <app_name>@instance_1 <app_name>@instance_2 ... <app_name>@instance_<N>

Deploying archived applications

While the RPM package places your application to /usr/share/tarantool/<app_name> on your server by default, the tar + gz archive does not enforce any structure apart from just the <app_name>/ directory, so you are responsible for placing it appropriately.

Note

RPM packages are recommended for deployment. Deploy archives only if you do not have root privileges.

To place and deploy the application, do the following on every server dedicated for Tarantool Enterprise:

  1. Take the tarantoolctl binary from the SDK and upload it anywhere on the server. If you do not have root privileges, a good place is $HOME/bin. Add this path to your ~/.bash_profile:

    $ export PATH="$HOME/bin:$PATH"
    
  2. Configure tarantoolctl to look for Tarantool instances in a certain place. For example, add the following lines to ~/.config/tarantool/tarantool:

    username = "<user>"
    instance_dir = "/home/<user>/apps"
    vinyl_dir = "/home/<user>/tarantool_state/var/lib/tarantool"
    memtx_dir = "/home/<user>/tarantool_state/var/lib/tarantool"
    snap_dir = "/home/<user>/tarantool_state/var/lib/tarantool"
    wal_dir = "/home/<user>/tarantool_state/var/lib/tarantool"
    log = "/home/<user>/tarantool_state/var/log/tarantool"
    pid_file = "/home/<user>/tarantool_state/var/run/tarantool"
    

    where <user> is the username of an account without root privileges, the instance_dir is the directory to unpack applications to, and the rest are state directories.

  3. Upload the archive, decompress, and extract it to the /home/<user>/apps directory:

    $ tar -xzvf <app_name>-<version>.tar.gz -C /home/<user>/apps
    
  4. Start one or multiple Tarantool instances with the corresponding services as described below.

    • A single instance:

      $ tarantoolctl start <app_name>
      
    • Multiple instances on one or multiple servers:

      $ tarantoolctl start <app_name>@instance_1
      $ tarantoolctl start <app_name>@instance_2
      ...
      $ tarantoolctl start <app_name>@instance_<number>
      

      where <number> is the incremental number (unique for every instance) to be added to the 3300 port the instance will listen to (e.g., 3301, 3302, etc.).

      This starts several instances from the same directory /home/<user>/apps/<app_name> but their state files differ depending on the suffix after @. For example, the log file of the first instance will have the following path:

      /home/<user>/tarantool_state/var/log/tarantool/<app_name>.instance_1.log
      
  5. In case it is a cluster-aware application, proceed to deploying the cluster.

To stop all instances on a server, run the following command for every instance:

$ tarantoolctl stop <app_name>[@instance_<number>]

Upgrading code

All instances in the cluster are to run the same code. This includes all the components: custom roles, applications, module binaries, tarantool and tarantoolctl (if necessary) executables.

Pay attention to possible backward incompatibility that any component may introduce. This will help you choose a scenario for an upgrade in production. Keep in mind that you are responsible for code compatibility and handling conflicts should inconsistencies occur.

To upgrade any of the components, prepare a new version of the package (archive):

  1. Update the necessary files in your development environment (directory):
    • Your own source code: custom roles and/or applications.
    • Module binaries.
    • Executables. Replace them with ones from the new bundle.
  2. Increment the version as described in application versioning.
  3. Repack the updated files as described in packaging applications.
  4. Choose an upgrade scenario as described in production upgrade section.

Running sample applications

The distribution package includes sample applications in the example/ directory that showcase basic Tarantool functionality.

Cluster application

The example in the cluster/ directory showcases a simple cluster-aware application. It consists of the following files:

  • init.lua – module containing the cluster.cfg() initialization function.
  • app.lua – role module to contain your stored procedures and API calls.
  • storage.lua – role module defining functions for the database.
  • deps.sh – script to resolve rocks dependencies from an offline repository included in the archive.
  • start.sh – script to start several Tarantool instances.
  • assemble.sh – script to assemble the cluster.
  • test.py – health check test showing how to put and get data from the running application.
  • clean.sh – script to clean the data.

Look through the code in the files to get an understanding of what the application does.

To start the sample application, do the following:

  1. Start several instances:

    $ ./start.sh
    
  2. Assemble the cluster by assigning roles to instances in the web interface or simply run:

    $ ./assemble.sh
    

To check if all the instances are up and running, say:

$ ps x|grep tarantool

To run a basic sanity check on the application, use the provided test script:

$ python test.py

To clean the data after, say:

$ ./clean.sh

To stop the application, say:

$ ./stop.sh

Write-through cache application for PostgreSQL

The example in pg_writethrough_cache/ shows how Tarantool can cache data written through it to a PostgreSQL database to speed up the reads.

The sample application requires a deployed PostgreSQL database and the following rock modules:

$ tarantoolctl rocks install http
$ tarantoolctl rocks install pg
$ tarantoolctl rocks install argparse

Look through the code in the files to get an understanding of what the application does.

To run the application for a local PostgreSQL database, say:

$ tarantool cachesrv.lua --binary-port 3333 --http-port 8888 --database postgresql://localhost/postgres

Write-behind cache application for Oracle

The example in ora-writebehind-cache/ shows how Tarantool can cache writes and queue them to an Oracle database to speed up both writes and reads.

Application requirements

The sample application requires:

  • deployed Oracle database;

  • Oracle tools: Instant Client and SQL Plus, both of version 12.2;

    Note

    In case the Oracle Instant Client errors out on .so files (Oracle’s dynamic libraries), put them to some directory and add it to the LD_LIBRARY_PATH environment variable.

    For example: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/<path_to_so_files>

  • rock modules listed in the rockspec file.

To install the modules, run the following command in the examples/ora_writebehind_cache directory:

$ tarantoolctl rocks make oracle_rb_cache-0.1.0-1.rockspec

If you do not have a deployed Oracle instance at hand, run a dummy in a Docker container:

  1. In browser, log in to Oracle container registry, click Database, and accept the Oracle’s Enterprise Terms and Restrictions.

  2. In the ora-writebehind-cache/ directory, log in to the repository under the Oracle account, pull, and run an image using the prepared scripts:

    $ docker login container-registry.oracle.com
    Login:
    Password:
    Login Succeeded
    $ docker pull container-registry.oracle.com/database/enterprise:12.2.0.1
    $ docker run -itd \
       -p 1521:1521 \
       -p 5500:5500 \
       --name oracle \
       -v "$(pwd)"/setupdb/configDB.sh:/home/oracle/setup/configDB.sh \
       -v "$(pwd)"/setupdb/runUserScripts.sh:/home/oracle/setup/runUserScripts.sh \
       -v "$(pwd)"/startupdb:/opt/oracle/scripts/startup \
       container-registry.oracle.com/database/enterprise:12.2.0.1
    

When all is set and done, run the example application.

Running write-behind cache

To launch the application, run the following in the examples/ora_writebehind_cache directory:

$ tarantool init.lua

The application supports the following requests:

  • Get: GET http://<host>:8080/account/id;

  • Add: POST http://<host>:8080/account/ with the following data:

    {"clng_clng_id":1,"asut_asut_id":2,"creation_data":"01-JAN-19","navi_user":"userName"}
    
  • Update: POST http://<host>:8080/account/id with the same data as in the add request;

  • Remove: DELETE http://<host>:8080/account/id where id is an account identifier.

Look for sample CURL scripts in the examples/ora_writebehind_cache/testing directory and check the README.md for more information on implementation.

Hello-world application in Docker

The example in the docker/ directory contains a hello-world application that you can pack in a Docker container and run on CentOS 7.

The hello.lua file is the entry point and it is very bare-bones, so you can add your own code here.

  1. To build the container, say:

    $ docker build -t tarantool-enterprise-docker -f Dockerfile ../..
    
  2. To run it:

    $ docker run --rm -t -i tarantool-enterprise-docker
    

Cluster administrator’s guide

This guide explains how to deploy and manage the Tarantool Enterprise cluster. For more information on managing Tarantool instances, see the Tarantool User’s Guide.

Before deploying the cluster, familiarize yourself with the notion of cluster roles and deploy Tarantool instances according to the desired cluster topology.

Deploying the cluster

To deploy the cluster, first, configure all the necessary replica sets according to the desired cluster topology, then bootstrap the cluster.

You can do this via the Web interface which is available at http://<instance_hostname>:<instance_http_port> (e.g., http://localhost:8081).

In the web interface, do the following:

  1. Depending on the authentication state:

    • If enabled (in production), enter your credentials and click Log in:

      _images/auth_creds.png

      And proceed to configuring the cluster.

    • If disabled (for easier testing), simply proceed to configuring the cluster.

      Note

      To enable it manually, click Log in in the upper left corner, enter the credentials, click another Log in and the Auth: disabled switch (it turns into Auth: enabled).

  2. Click Create next to the first unconfigured server to create the first replica set solely for the router.

    _images/unconfigured-router.png
  3. In the pop-up window, check the vshard-router role.

    _images/create-router.png

    Additionally, check every custom cluster role meant to handle compute-intensive workloads.

    Important

    As described in the built-in roles section, it is a good practice to enable workload-specific cluster roles on instances running on physical servers with workload-specific hardware.

    And click Submit.

  4. (Optional) If required by topology, populate the newly created replica set with more routers (and compute nodes in parallel):

    1. Click Join next to other unconfigured instances:

      _images/join-router.png
    2. Select the first router (and compute node) and click Submit:

      _images/select-router.png

      The current instance will inherit cluster roles from the replica set.

  5. Create another replica set for storage nodes by repeating the first step and checking the vshard-storage role.

    _images/create-storage.png

    Additionally, check every custom cluster role meant to handle transaction-intensive workloads.

    And click Submit.

  6. Click Join next to another unconfigured server dedicated for transaction-intensive workloads.

    _images/join-replicaset.png
  7. Select the second (vshard-storage) replica set, and click Submit to add the server to it.

    _images/join-storage.png
  8. Depending on cluster topology:

    • add more instances to the first or second sets, or
    • create a new one and populate it with instances meant to handle either compute or transactions.

    For example:

    _images/final-cluster.png
  9. (Optional) By default, all new vshard-storage replica sets get a weight of 1 before the vshard’s bootstrap in the next step.

    Note

    In case you add a new replica set after the bootstrap as described in the topology change section, it will get a weight of 0 by default.

    To make different replica sets store different numbers of buckets, click Edit next to a replica set, change its default weight, and click Submit:

    _images/change-weight.png

    For more information on buckets and replica set’s weights, see the vshard module documentation.

  10. Bootstrap vshard by clicking the corresponding button and OK, or by saying cluster.admin.boostrap_vshard() over the administrative console.

    _images/bootstrap-vshard.png

    This command creates virtual buckets and distributes them among storages.

From now on, all the necessary cluster configuration can be done via the web interface.

Automatic configuration synchronization

All instances in Tarantool cluster have the same configuration. To this end, every instance stores a copy of the configuration file. The cluster keeps these copies in sync.

The distributed configuration file is changed automatically upon submits in the Web interface and includes cluster topology and role descriptions.

The cluster validates any configuration change and rejects inappropriate ones.

Managing the cluster

This chapter explains how to:

  • change the cluster topology,
  • enable automatic failover,
  • switch the replica set’s master manually,
  • deactivate replica sets, and
  • expel instances.

Changing the cluster topology

Upon adding a newly deployed instance to a new or existing replica set:

  1. The cluster validates the configuration update by checking if the new instance is available using the membership module.

    Note

    The membership module works over the UDP protocol and can operate before the box.cfg function is called.

    All the nodes in the cluster must be healthy for validation success.

  2. The new instance waits until another instance in the cluster receives the configuration update and discovers it, again, using the membership module. On this step, the new instance does not have a UUID yet.

  3. Once the instance realizes its presence is known to the cluster, it calls the box.cfg function and starts living its life.

    For more information, see the box.cfg submodule reference.

An optimal strategy for connecting new nodes to the cluster is to deploy a new zero-weight replica set instance by instance, and then increase the weight. Once the weight is updated and all cluster nodes are notified of the configuration change, buckets start migrating to new nodes.

To populate the cluster with more nodes, do the following:

  1. Deploy new Tarantool instances as described in the deployment section.

    If new nodes do not appear in the Web interface, click Probe server and specify their URIs manually.

    _images/probe-server.png
  2. In the Web interface, click Create to add an unconfigured instance to a new replica set:

    _images/new-unconfig.png

    Check the necessary roles and click Submit.

    Note

    In case you are adding a new vshard-storage instance, remember that all such instances get a 0 weight by default after the vshard’s bootstrap which happened during the initial cluster deployment.

    _images/zero.png
  3. Click Join next to another unconfigured server, select the new replica set, and click Submit.

    _images/join-new-set.png
  4. If necessary, repeat the above step for more instances to reach the desired redundancy level.

  5. In case you are deploying a new vshard-storage replica set, populate it with data when you are ready.

    _images/change-weight.png

    Click Edit next to the replica set in question, increase its weight, and click Submit to start data rebalancing.

Data rebalancing

Rebalancing (resharding) is initiated periodically and upon adding a new replica set with a non-zero weight to the cluster. For more information, see the rebalancing process section of the vshard module documentation.

The most convenient way to trace through the process of rebalancing is to monitor the number of active buckets on storage nodes. Initially, a newly added replica set has 0 active buckets. After a few minutes, the background rebalancing process begins to transfer buckets from other replica sets to the new one. Rebalancing continues until the data is distributed evenly among all replica sets.

To monitor the current number of buckets, connect to any Tarantool instance over the administrative console, and say:

$ tarantool> vshard.storage.info().bucket
---
- receiving: 0
  active: 1000
  total: 1000
  garbage: 0
  sending: 0
...

The number of buckets may be increasing or decreasing depending on whether the rebalancer is migrating buckets to or from the storage node.

For more information on the monitoring parameters, see the monitoring storages section.

Deactivating replica sets

To deactivate an entire replica set (e.g., to perform maintenance on it) means to move all of its buckets to other sets.

To deactivate a set, do the following:

  1. Click Edit next to the set in question.

  2. Set its weight to 0 and click Submit:

    _images/zero-weight.png
  3. Wait for the rebalancing process to finish migrating all the set’s buckets away. You can monitor the current bucket number as described in the data rebalancing section.

Expelling instances

Once an instance is expelled, it can never participate in the cluster again as every instance will reject it.

To expel an instance, click Expel next to it, then OK:

_images/expelling-instance.png

Enabling automatic failover

In a master-replica cluster configuration with automatic failover enabled, if the user-specified master of any replica set fails, the cluster automatically chooses the next replica from the priority list and grants it the active master role. When the failed master comes back online, its role is restored and the active master, again, becomes a replica.

To set the priority in a replica set, click Edit next to the set in question, drag replicas to their place in the priority list, and click Submit:

_images/failover-priority.png

The failover is disabled by default. To enable it, click Failover: disabled:

_images/failover.png

And, in the Failover control window, click Enable:

_images/failover-control.png

The failover status will change to enabled:

_images/enabled-failover.png

For more information, see the replication section of the Tarantool manual.

Switching the replica set’s master

To manually switch the master in a replica set, click the Edit button next to the replica set in question:

_images/edit-replica-set.png

Select another master and click Submit:

_images/switch-master.png

Exploring spaces

The web interface lets you connect (in the browser) to any instance in the cluster and see what spaces it stores (if any) and their contents.

To explore spaces:

  1. Open the Space Explorer tab in the menu on the left:

    _images/space_explr_tab.png

    The instances you see were deployed in scope of the example cluster application.

  2. Click connect next to an instance that stores data. The basic sanity-check (test.py) of the example application puts sample data to one replica set (shard), so its master and replica store the data in their spaces:

    _images/spaces_with_data.png

    When connected to a instance, the space explorer shows a table with basic information on its spaces. For more information, see the box.space reference.

    To see hidden spaces, tick the corresponding checkbox:

    _images/hidden_spaces.png
  3. Click the space’s name to see its format and contents:

    _images/space_contents.png

    To search the data, select an index and, optionally, its iteration type from the drop-down lists, and enter the index value:

    _images/space_search.png

Resolving conflicts

Tarantool has an embedded mechanism for asynchronous replication. As a consequence, records are distributed among the replicas with a delay, so conflicts can arise.

To prevent conflicts, the special trigger space.before_replace is used. It is executed every time before making changes to the table for which it was configured. The trigger function is implemented in the Lua programming language. This function takes the original and new values of the tuple to be modified as its arguments. The returned value of the function is used to change the result of the operation: this will be the new value of the modified tuple.

For insert operations, the old value is absent, so nil is passed as the first argument.

For delete operations, the new value is absent, so nil is passed as the second argument. The trigger function can also return nil, thus turning this operation into delete.

This example shows how to use the space.before_replace trigger to prevent replication conflicts. Suppose we have a box.space.test table that is modified in multiple replicas at the same time. We store one payload field in this table. To ensure consistency, we also store the last modification time in each tuple of this table and set the space.before_replace trigger, which gives preference to newer tuples. Below is the code in Lua:

fiber = require('fiber')
-- define a function that will modify the function test_replace(tuple)
        -- add a timestamp to each tuple in the space
        tuple = box.tuple.new(tuple):update{{'!', 2, fiber.time()}}
        box.space.test:replace(tuple)
end
box.cfg{ } -- restore from the local directory
-- set the trigger to avoid conflicts
box.space.test:before_replace(function(old, new)
        if old ~= nil and new ~= nil and new[2] < old[2] then
                return old -- ignore the request
        end
        -- otherwise apply as is
end)
box.cfg{ replication = {...} } -- subscribe

Upgrading in production

To upgrade either a single instance or a cluster, you need a new version of the packaged (archived) application.

A single instance upgrade is simple:

  1. Upload the package (archive) to the server.
  2. Stop the current instance.
  3. Deploy the new one as described in deploying packaged applications (or archived ones).

Cluster upgrade

To upgrade a cluster, choose one of the following scenarios:

  • Cluster shutdown. Recommended for backward-incompatible updates, requires downtime.
  • Instance by instance. Recommended for backward-compatible updates, does not require downtime.

To upgrade the cluster, do the following:

  1. Schedule a downtime or plan for the instance-by-instance upgrade.
  2. Upload a new application package (archive) to all servers.

Next, execute the chosen scenario:

  • Cluster shutdown:
    1. Stop all instances on all servers.
    2. Deploy the new package (archive) on every server.
  • Instance by instance. Do the following in every replica set in succession:
    1. Stop a replica on any server.
    2. Deploy the new package (archive) in place of the old replica.
    3. Promote the new replica to a master as described in switching the master section.
    4. Redeploy the old master and the rest of the instances in the replica set.
    5. Be prepared to resolve possible logic conflicts.

Monitoring cluster via CLI

This section describes parameters you can monitor over the administrative console.

Connecting to nodes via CLI

Each Tarantool node (router/storage) provides an administrative console (Command Line Interface) for debugging, monitoring, and troubleshooting. The console acts as a Lua interpreter and displays the result in the human-readable YAML format. To connect to a Tarantool instance via the console, say:

$ tarantoolctl connect <instance_hostname>:<port>

where the <instance_hostname>:<port> is the instance’s URI.

Monitoring storages

Use vshard.storage.info() to obtain information on storage nodes.

Output example

$ tarantool> vshard.storage.info()
---
- replicasets:
    <replicaset_2>:
    uuid: <replicaset_2>
    master:
        uri: storage:storage@127.0.0.1:3303
    <replicaset_1>:
    uuid: <replicaset_1>
    master:
        uri: storage:storage@127.0.0.1:3301
  bucket: <!-- buckets status
    receiving: 0 <!-- buckets in the RECEIVING state
    active: 2 <!-- buckets in the ACTIVE state
    garbage: 0 <!-- buckets in the GARBAGE state (are to be deleted)
    total: 2 <!-- total number of buckets
    sending: 0 <!-- buckets in the SENDING state
  status: 1 <!-- the status of the replica set
  replication:
    status: disconnected <!-- the status of the replication
    idle: <idle>
  alerts:
  - ['MASTER_IS_UNREACHABLE', 'Master is unreachable: disconnected']

Status list

Code Critical level Description
0 Green A replica set works in a regular way.
1 Yellow There are some issues, but they don’t affect a replica set efficiency (worth noticing, but don’t require immediate intervention).
2 Orange A replica set in in a degraded state.
3 Red A replica set is disabled.

Potential issues

  • MISSING_MASTER — No master node in the replica set configuration.

    Critical level: Orange.

    Cluster condition: Service is degraded for data-change requests to the replica set.

    Solution: Set the master node for the replica set in the configuration using API.

  • UNREACHABLE_MASTER — No connection between the master and the replica.

    Critical level:

    • If idle value doesn’t exceed T1 threshold (1 s.) — Yellow,
    • If idle value doesn’t exceed T2 threshold (5 s.) — Orange,
    • If idle value exceeds T3 threshold (10 s.) — Red.

    Cluster condition: For read requests to replica, the data may be obsolete compared with the data on master.

    Solution: Reconnect to the master: fix the network issues, reset the current master, switch to another master.

  • LOW_REDUNDANCY — Master has access to a single replica only.

    Critical level: Yellow.

    Cluster condition: The data storage redundancy factor is equal to 2. It is lower than the minimal recommended value for production usage.

    Solution: Check cluster configuration:

    • If only one master and one replica are specified in the configuration, it is recommended to add at least one more replica to reach the redundancy factor of 3.
    • If three or more replicas are specified in the configuration, consider checking the replicas’ states and network connection among the replicas.
  • INVALID_REBALANCING — Rebalancing invariant was violated. During migration, a storage node can either send or receive buckets. So it shouldn’t be the case that a replica set sends buckets to one replica set and receives buckets from another replica set at the same time.

    Critical level: Yellow.

    Cluster condition: Rebalancing is on hold.

    Solution: There are two possible reasons for invariant violation:

    • The rebalancer has crashed.
    • Bucket states were changed manually.

    Either way, please contact Tarantool support.

  • HIGH_REPLICATION_LAG — Replica’s lag exceeds T1 threshold (1 sec.).

    Critical level:

    • If the lag doesn’t exceed T1 threshold (1 sec.) — Yellow;
    • If the lag exceeds T2 threshold (5 sec.) — Orange.

    Cluster condition: For read-only requests to the replica, the data may be obsolete compared with the data on the master.

    Solution: Check the replication status of the replica. Further instructions are given in the :ref: troubleshooting guide <admin-troubleshoot>`.

  • OUT_OF_SYNC — Mal-synchronization occured. The lag exceeds T3 threshold (10 sec.).

    Critical level: Red.

    Cluster condition: For read-only requests to the replica, the data may be obsolete compared with the data on the master.

    Solution: Check the replication status of the replica. Further instructions are given in the :ref: troubleshooting guide <admin-troubleshoot>`.

  • UNREACHABLE_REPLICA — One or multiple replicas are unreachable.

    Critical level: Yellow.

    Cluster condition: Data storage redundancy factor for the given replica set is less than the configured factor. If the replica is next in the queue for rebalancing (in accordance with the weight configuration), the requests are forwarded to the replica that is still next in the queue.

    Solution: Check the error message and find out which replica is unreachable. If a replica is disabled, enable it. If this doesn’t help, consider checking the network.

  • UNREACHABLE_REPLICASET — All replicas except for the current one are unreachable. Critical level: Red.

    Cluster condition: The replica stores obsolete data.

    Solution: Check if the other replicas are enabled. If all replicas are enabled, consider checking network issues on the master. If the replicas are disabled, check them first: the master might be working properly.

Monitoring routers

Use vshard.router.info() to obtain information on the router.

Output example

$ tarantool> vshard.router.info()
---
- replicasets:
    <replica set UUID>:
      master:
        status: <available / unreachable / missing>
        uri: <!-- URI of master
        uuid: <!-- UUID of instance
      replica:
        status: <available / unreachable / missing>
        uri: <!-- URI of replica used for slave requests
        uuid: <!-- UUID of instance
      uuid: <!-- UUID of replica set
    <replica set UUID>: ...
    ...
  status: <!-- status of router
  bucket:
    known: <!-- number of buckets with the known destination
    unknown: <!-- number of other buckets
  alerts: [<alert code>, <alert description>], ...

Status list

Code Critical level Description
0 Green The router works in a regular way.
1 Yellow Some replicas sre unreachable (affects the speed of executing read requests).
2 Orange Service is degraded for changing data.
3 Red Service is degraded for reading data.

Potential issues

Note

Depending on the nature of the issue, use either the UUID of a replica, or the UUID of a replica set.

  • MISSING_MASTER — The master in one or multiple replica sets is not specified in the configuration.

    Critical level: Orange.

    Cluster condition: Partial degrade for data-change requests.

    Solution: Specify the master in the configuration.

  • UNREACHABLE_MASTER — The router lost connection with the master of one or multiple replica sets.

    Critical level: Orange.

    Cluster condition: Partial degrade for data-change requests.

    Solution: Restore connection with the master. First, check if the master is enabled. If it is, consider checking the network.

  • SUBOPTIMAL_REPLICA — There is a replica for read-only requests, but this replica is not optimal according to the configured weights. This means that the optimal replica is unreachable.

    Critical level: Yellow.

    Cluster condition: Read-only requests are forwarded to a backup replica.

    Solution: Check the status of the optimal replica and its network connection.

  • UNREACHABLE_REPLICASET — A replica set is unreachable for both read-only and data-change requests.

    Critical Level: Red.

    Cluster condition: Partial degrade for read-only and data-change requests.

    Solution: The replica set has an unreachable master and replica. Check the error message to detect this replica set. Then fix the issue in the same way as for UNREACHABLE_REPLICA.

Troubleshooting

Please see the Troubleshooting guide in the Tarantool manual.

Disaster recovery

Please see the section Disaster recovery in the Tarantool manual.

Backups

Please see the section Backups in the Tarantool manual.

Reporting issues

To report a bug or submit an update request for this document, please create an issue in our repository at GitLab.

Security hardening guide

This guide explains how to enhance security in your Tarantool Enterprise cluster using built-in features and provides general recommendations on security hardening.

Tarantool Enterprise does not provide a dedicated API for security control. All the necessary configuration can be done via an administrative console or initialization code.

Built-in security features

Tarantool Enterprise has the following built-in security features:

  • authentication,
  • access control,
  • audit log;

And backup functionality:

The following sections describe the features and provide links to detailed documentation on each.

Authentication

Tarantool Enterprise supports password-based authentication and allows for two types of connections:

  • via an administrative console and
  • over a binary port for read and write operations and procedure invocation.

For more information on authentication and connection types, see the security section of the Tarantool manual.

In addition, Tarantool provides the following functionality:

  • sessions – states which associate connections with users and make Tarantool API available to them after authentication,
  • authentication triggers which execute actions on authentication events.

Tarantool Enterprise does not support third-party (external) authentication protocols or services such as LDAP or Active Directory.

Access control

Tarantool Enterprise provides the means for administrators to prevent unauthorized access to the database and to certain functions.

Tarantool recognizes:

  • different users (guests and administrators),
  • privileges associated with users,
  • roles (containers for privileges) granted to users;

And divides system space into:

  • _user space to store usernames and hashed passwords for authentication,
  • _priv space to store privileges for access control.

For more information, see the access control section of the Tarantool manual.

Users who create objects (spaces, indexes, users, roles, sequences, and functions) in the database become their owners and automatically acquire privileges for what they create. For more information, see the owners and privileges section of the Tarantool manual.

Audit log

Tarantool Enterprise has a built-in audit log that records events such as:

  • authentication successes and failures;
  • connection closures;
  • creation, removal, enabling, and disabling of users;
  • changes of passwords, privileges, and roles;
  • denials of access to database objects;

And contains:

  • timestamps,
  • usernames of users who performed actions,
  • event types (e.g. user_create, user_enable, disconnect, etc),
  • descriptions.

Audit log has two configuration parameters:

  • audit_log = <PATH_TO_FILE> which tells Tarantool to record audit events to a specific file and is similar to the log parameter,
  • and audit_nonblock which is similar to the log_nonblock parameter.

For more information on logging, see the following:

Access permissions to audit log files can be set up as to any other Unix file system object – via chmod.

Recommendations on security hardening

This section lists recommendations that can help you harden the cluster’s security.

Traffic encryption

Tarantool Enterprise does not encrypt traffic over binary connections (i.e., between servers in the cluster). To secure such connections, consider:

  • setting up connection tunneling, or
  • encrypting the actual data stored in the database.

For more information on data encryption, see the crypto module reference.

The HTTP server module provided by rocks does not support the HTTPS protocol. To set up a secure connection for a client (e.g., REST service), consider hiding the Tarantool instance (router if it is a cluster of instances) behind a Nginx server and setting up an SSL certificate for it.

To make sure that no information can be intercepted ‘from the wild’, run Nginx on the same physical server as the instance and set up their communication over a Unix socket. For more information, see the socket module reference.

Firewall configuration

To protect the cluster from any unwanted network activity ‘from the wild’, configure the firewall on each server to allow traffic on ports listed in network requirements.

If you are using static IP addresses, whitelist them, again, on each server as the cluster has a full mesh network topology. Consider blacklisting all the other addresses on all servers except the router (running behind the Nginx server).

Tarantool Enterprise does not provide defense against DoS or DDoS attacks. Consider using third-party software instead.

Data integrity

Tarantool Enterprise does not keep checksums or provide the means to control data integrity. However, it ensures data persistence using a write ahead log, regularly snapshots the entire data set to disk, and checks the data format whenever it reads the data back from the disk. For more information, see the data persistence section of the Tarantool manual.

Rocks reference

This chapter covers open and closed source Lua modules for Tarantool Enterprise included in the distribution as an offline rocks repository.

Open source modules:

  • vshard is an automatic sharding system for Tarantool DBMS instances;
  • ldecnumber is the decimal arithmetic library;
  • http is an on-board HTTP-server;
  • stat is a selection of useful monitoring metrics;
  • checks is a type checker of functional arguments;
  • luarapidxml is a fast XML parser;
  • avro-schema is an assembly of Apache Avro schema tools.

Closed source modules:

  • cluster is a high-level cluster management interface that contains several modules:
    • rpc implements remote procedure calls between cluster instances and allows roles running on some instances to interact with other roles on other instances.
    • service-registry implements inter-role interaction and allows different roles to interact with each other in the scope of one instance.
    • confapplier implements cluster-wide configuration validation and application via a two-phase commit.
    • auth manages authentication.
    • pool reuses Tarantool’s net.box connections.
    • admin implements administration functions.
  • membership is a mesh of instances based on gossip protocol;
  • oracle is an Oracle connector for Lua applications though which they can access Oracle databases.

Installing and using modules

To use a module, install the following:

  1. All the necessary third-party software packages (if any). See the module’s prerequisites for the list.

  2. The module itself on every Tarantool instance:

    $ tarantoolctl rocks install <module_name> [<module_version>]
    
  3. Consult the module’s API documentation:

    $ tarantoolctl rocks doc <module-name>
    

    Or read the module’s reference to get a grasp on how to use it:

Module membership

This module is a membership library for Tarantool based on a gossip protocol.

This library builds a mesh from multiple Tarantool instances. The mesh monitors itself, helps members discover everyone else in the group and get notified about their status changes with low latency. It is built upon the ideas from Consul or, more precisely, the SWIM algorithm.

The membership module works over UDP protocol and can operate even before the box.cfg initialization.

Member data structure

A member is represented by the table with the following fields:

  • uri (string) is a Uniform Resource Identifier.

  • status (string) is a string that takes one of the values below.

    • alive: a member that replies to ping-messages is alive and well.

    • suspect: if any member in the group cannot get a reply from any other member, the first member asks three other alive members to send a ping-message to the member in question. If there is no response, the latter becomes a suspect.

    • dead: a suspect becomes dead after a timeout.

    • left: a member gets the left status after executing the leave() function.

      Note

      The gossip protocol guarantees that every member in the group becomes aware of any status change in two communication cycles.

  • incarnation (number) is a value incremented every time the instance is becomes a suspect, dead, or updates its payload.

  • payload (table) is auxiliary data that can be used by various modules.

  • timestamp (number) is a value of fiber.time64() which:

    • corresponds to the last update of status or incarnation;
    • is always local;
    • does not depend on other members’ clock setting.

Below is an example of the table:

tarantool> membership.myself()
---
uri: localhost:33001
status: alive
incarnation: 1
payload:
    uuid: 2d00c500-2570-4019-bfcc-ab25e5096b73
timestamp: 1522427330993752
...

API reference

Below is a list of membership’s common, encryption, subscription functions, and options.

Name Use
Common functions
init(advertise_host, port) Initialize the membership module.
myself() Get the member data structure of the current instance.
get_member(uri) Get the member data structure for a given URI.
members() Obtain a table with all members known to the current instance.
pairs() Shorthand for pairs(membership.members()).
add_member(uri) Add a member to the group.
probe_uri(uri) Check if the member is in the group.
broadcast() Discover members in LAN by sending a UDP broadcast message.
set_payload(key, value) Update myself().payload and disseminate it.
leave() Gracefully leave the group.
is_encrypted() Check if encryption is enabled.
Encryption functions
set_encryption_key(key) Set the key for low-level message encryption.
get_encryption_key() Retrieve the encryption key in use.
Subscription functions
subscribe() Subscribe for the members table updates.
unsubscribe() Remove the subscription.
Options
PROTOCOL_PERIOD_SECONDS Direct ping period.
ACK_TIMEOUT_SECONDS ACK message wait time.
ANTI_ENTROPY_PERIOD_SECONDS Anti-entropy synchronization period.
SUSPECT_TIMEOUT_SECONDS Timeout to mark a suspect dead.
NUM_FAILURE_DETECTION_SUBGROUPS Number of members to ping a suspect indirectly.

Common functions:

membership.init(advertise_host, port)

Initialize the membership module. This binds a UDP socket to 0.0.0.0:<port>, sets the advertise_uri parameter to <advertise_host>:<port>, and incarnation to 1.

The init() function can be called several times, the old socket will be closed and a new one opened.

If the advertise_uri changes during the next init(), the old URI is considered DEAD. In order to leave the group gracefully, use the leave() function.

Parameters:
  • advertise_host (string) – a hostname or IP address to advertise to other members
  • port (number) – a UDP port to bind
Returns:

true

Return type:

boolean

Raises:

socket bind error

membership.myself()
Returns:the member data structure of the current instance.
Return type:table
membership.get_member(uri)
Parameters:uri (string) – the given member’s advertise_uri
Returns:the member data structure of the instance with the given URI.
Return type:table
membership.members()

Obtain all members known to the current instance.

Editing this table has no effect.

Returns:a table with URIs as keys and corresponding member data structures as values.
Return type:table
membership.pairs()

A shorthand for pairs(membership.members()).

Returns:Lua iterator

It can be used in the following way:

for uri, member in memberhip.pairs()
  -- do something
end
membership.add_member(uri)

Add a member with the given URI to the group and propagate this event to other members. Adding a member to a single instance is enough as everybody else in the group will receive the update with time. It does not matter who adds whom.

Parameters:uri (string) – the advertise_uri of the member to add
Returns:true or nil in case of an error
Return type:boolean
Raises:parse error if the URI cannot be parsed
membership.probe_uri(uri)

Send a message to a member to make sure it is in the group. If the member is alive but not in the group, it is added. If it already is in the group, nothing happens.

Parameters:uri (string) – the advertise_uri of the member to ping
Returns:true if the member responds within 0.2 seconds, otherwise no response
Return type:boolean
Raises:ping was not sent if the hostname could not be resolved
membership.broadcast()

Discover members in local network by sending a UDP broadcast message to all networks discovered by a getifaddrs() C call.

Returns:true if broadcast was sent, false if getaddrinfo() fails.
Return type:boolean
membership.set_payload(key, value)

Update myself().payload and disseminate it along with the member status.

Increments incarnation.

Parameters:
  • key (string) – a key to set in payload table
  • value – auxiliary data
Returns:

true

Return type:

boolean

membership.leave()

Gracefully leave the membership group. The node will be marked with the left status and no other members will ever try to reconnect it.

Returns:true
Return type:boolean
membership.is_encrypted()
Returns:true if encryption is enabled, false otherwise.
Return type:boolean

Encryption functions:

membership.set_encryption_key(key)

Set the key used for low-level message encryption. The key is either trimmed or padded automatically to be exactly 32 bytes. If the key value is nil, the encryption is disabled.

The encryption is handled by the crypto.cipher.aes256.cbc Tarantool module.

For proper communication, all members must be configured to use the same encryption key. Otherwise, members report either dead or non-decryptable in their status.

Parameters:key (string) – encryption key
Returns:nil.
membership.get_encryption_key()

Retrieve the encryption key that is currently in use.

Returns:encryption key or nil if the encryption is disabled.
Return type:string

Subscription functions:

membership.subscribe()

Subscribe for updates in the members table.

Returns:a fiber.cond object broadcasted whenever the members table changes.
Return type:object
membership.unsubscribe(cond)

Remove subscription on cond obtained by the subscribe() function.

The cond’s validity is not checked.

Parameters:cond – the fiber.cond object obtained from subscribe()
Returns:nil.

Below is a list of membership options. They can be set as follows:

options = require('membership.options')
options.<option> = <value>
options.PROTOCOL_PERIOD_SECONDS

Period of sending direct pings. Denoted as T' in the SWIM protocol.

options.ACK_TIMEOUT_SECONDS

Time to wait for ACK message after a ping. If a member is late to reply, the indirect ping algorithm is invoked.

options.ANTI_ENTROPY_PERIOD_SECONDS

Period to perform the anti-entropy synchronization algorithm of the SWIM protocol.

options.SUSPECT_TIMEOUT_SECONDS

Timeout to mark suspect members as dead.

options.NUM_FAILURE_DETECTION_SUBGROUPS

Number of members to try pinging a suspect indirectly. Denoted as k in the SWIM protocol.

Module vshard

The vshard module introduces the sharding feature that enables horizontal scaling in Tarantool.

For more information, see the vshard module documentation.

Module http

In addition to the out-of-the-box HTTP client, this module includes an HTTP server but the latter has to be installed as described in the installation section.

For more information, see the HTTP README file in Tarantool’s GitHub repository.

Module checks

This is a library that declares a checks() function and checkers table that allow to check the parameters passed to a Lua function in a fast and unobtrusive way.

For more information, see the checks README file in Tarantool’s GitHub repository.

Module oracle

The oracle module is an Oracle connector for Tarantool. With this package, Lua applications can send and receive data to and from Oracle databases.

The advantage of the Tarantool-Oracle integration is that anyone can handle all the tasks with Oracle DBMSs (control, manipulation, storage, access) with the same high-level language (Lua) and with minimal delay.

Prerequisites

The oracle module requires the Instant Client Package (ICP) from Oracle. To install it via CLI, say:

curl -O https://raw.githubusercontent.com/bumpx/oracle-instantclient/master/oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
rpm -Uvh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm

Or get the package at oracle.com and install it the same way.

Note

Please, note the version is 12.2.

Quick start

Start Tarantool in the interactive mode and do the following:

  1. Connect to an Oracle database:

    tarantool> ora = require('oracle')
    tarantool> c, errmsg = ora.connect({username='system', password='oracle', db='localhost:1511/myspace'})
    tarantool> if not c then error("Failed to connect: "..errmsg) end
    
  2. Create a table and put some data into it:

    tarantool> c:exec('CREATE TABLE test(i int, s varchar(20))')
    tarantool> c:exec('INSERT INTO test(i, s) VALUES(1, \'Hello!\')')
    
  3. Retrieve the result:

    tarantool> rc, result = c:exec('SELECT * FROM test')
    tarantool> result[1]['S']
    ---
    'Hello!'
    ...
    

API reference

Below is a list of oracle functions.

Name Use
:exec(‘sql’) Execute an SQL request or raise an error.
:version() Get version string.
:exec('sql')

Executes an SQL request or raises an error.

Parameters:sql – SQL statement with the appropriate syntax.
Returns:
  • 0 in case of success; additionally, data and its size in case it was a SELECT request.
  • 1 in case of an ORA-error and the corresponding error message.

Examples

tarantool> c:exec('CREATE TABLE test(i int, s varchar(20))')
tarantool> c:exec('INSERT INTO test(i, s) VALUES(1, \'Hello!\')')
tarantool> c:exec('SELECT * FROM test')
:version()
Returns:versions of both the connector and OCI.
Return type:string

Example

tarantool> c:version()
---
- 'version: 1.1.0, OCI version(builded): 12.2, OCI version(linked): ?'
...

Appendixes

Appendix A. Audit log

Audit log provides records on the Tarantool DBMS events in the JSON-format. The following event logs are available:

  • successful/failed user authentication and authorization,
  • closed connection,
  • password change,
  • creation/deletion of a user/role,
  • enabling/disabling a user,
  • changing privileges of a user/role.

Log structure

Key Type Description Example
type string type of event
<“access_denied”>
type_id number id of event
<8>
description string description of event
<“Authentication failed”>
time string time of event
<“YYYY-MM-DDTHH:MM:SS.03f[+|-]GMT”>
peer string remote client
<“ip:port”>
user string user
<“user”>
param string parameters of event see below

Events description

Event Key Parameters
user authorized successfully auth_ok
{“name”: “user”}
user authorization failed auth_fail
{“name”: “user”}
user logged out or quit the session disconnect  
failed access attempts to secure data (personal records, details, geolocation, etc.) access_denied
{“name”: “obj_name”,
“obj_type“: “space”,
“access_type”: “read”}
creating a user user_create
{“name”: “user”}
dropping a user user_drop
{“name”: “user”}
disabling a user user_disable
{“name”: “user”}
enabling a user user_enable
{“name”: “user”}
granting (changing) privileges (roles, profiles, etc.) for the user user_priv
{“name”: “user”}
“obj_name”: “obj_name”,
“obj_type”: “space”,
“old_priv”: “”,
“new_priv”: “read,write”}
resetting password of the user (the user making changes should be specified) password_change
{“name”: “user”}
creating a role role_create
{“name”: “role”}
granting (changing) privileges for the role role_priv
{“name”: “role”}
“obj_name”: “obj_name”,
“obj_type”: “space”,
“old_priv”: “”,
“new_priv”: “read,write”}

Appendix B. Useful Tarantool parameters

  • box.info
  • box.info.replication
  • box.info.memory
  • box.stat
  • box.stat.net
  • box.slab.info
  • box.slab.stats

For details, please see reference on module `box` in the main Tarantool documentation.

Appendix C. Monitoring system metrics

Option Description SNMP type Units of measure Threshold
Version Tarantool version DisplayString    
IsAlive instance availability indicator

Integer

(listing)

 

0 - unavailable

1 - available

MemoryLua storage space used by Lua Gauge32 Mbyte 900
MemoryData storage space used for storing data Gauge32 Mbyte set the value manually
MemoryNet storage space used for network I/O Gauge32 Mbyte 1024
MemoryIndex storage space used for storing indexes Gauge32 Mbyte set the value manually
MemoryCache storage space used for storing caches (for vinyl engine only) Gauge32 Mbyte  
ReplicationLag lag time since the last sync between (the maximum value in case there are multiple fibers) Integer32 sec. 5
FiberCount number of fibers Gauge32 pc. 1000
CurrentTime current time, in seconds, starting at January, 1st, 1970 Unsigned32 Unix timestamp, in sec.  
StorageStatus status of a replica set Integer listing > 1
StorageAlerts number of alerts for storage nodes Gauge32 pc. >= 1
StorageTotalBkts total number of buckets in the storage Gauge32 pc. < 0
StorageActiveBkts number of buckets in the ACTIVE state Gauge32 pc. < 0
StorageGarbageBkts number of buckets in the GARBAGE state Gauge32 pc. < 0
StorageReceivingBkts number of buckets in the RECEIVING state Gauge32 pc. < 0
StorageSendingBkts number of buckets in the SENDING state Gauge32 pc. < 0
RouterStatus status of the router Integer listing > 1
RouterAlerts number of alerts for the router Gauge32 pc. >= 1
RouterKnownBkts number of buckets within the known destination replica sets Gauge32 pc. < 0
RouterUnknownBkts number of buckets that are unknown to the router Gauge32 pc. < 0
RequestCount total number of requests Counter64 pc.  
InsertCount total number of insert requests Counter64 pc.  
DeleteCount total number of delete requests Counter64 pc.  
ReplaceCount total number of replace requests Counter64 pc.  
UpdateCount total number of update requests Counter64 pc.  
SelectCount total number of select requests Counter64 pc.  
EvalCount number of calls made via Eval Counter64 pc.  
CallCount number of calls made via call Counter64 pc.  
ErrorCount number of errors in Tarantool Counter64 pc.  
AuthCount number of completed authentication operations Counter64 pc.  

Appendix D. Deprecated features

The ZooKeeper along with orchestrator are no longer supported. However, they still can be used, if necessary.

The following sections describe the corresponding functionality.

Controlling the cluster via API

To control the cluster, use the orchestrator included in the delivery package. The orchestrator uses ZooKeeper to store and distribute the configuration. The orchestrator provides the REST API for controlling the cluster. Configurations in the ZooKeeper are changed as a result of calling the orchestrator’s API-functions, which in turn leads to changes in configurations of the Tarantool nodes.

We recommend using a curl command line interface to call the API-functions of the orchestrator.

The following example shows how to register a new availability zone (DC):

$ curl -X POST http://HOST:PORT/api/v1/zone \
    -d '{
  "name": "Caucasian Boulevard"
  }'

To check whether the DC registration was successful, try the following instruction. It retrieves the list of all registered nodes in the JSON format:

$ curl http://HOST:PORT/api/v1/zone| python -m json.tool

To apply the new configuration directly on the Tarantool nodes, increase the configuration version number after calling the API function. To do this, use the POST request to /api/v1/version:

$ curl -X POST http://HOST:PORT/api/v1/version

Altogether, to update the cluster configuration:

  1. Call the POST/PUT method of the orchestrator.
    As a result, the ZooKeeper nodes are updated, and a subsequent update of the Tarantool nodes is initiated.
  2. Update the configuration version using the POST request to /api/v1/version.
    As a result, the configuration is applied to the Tarantool nodes.

See Appendix E for the detailed orchestrator API.

Setting up geo redundancy

Logically, cluster nodes can belong to some availability zone. Physically, an availability zone is a separate DC, or a rack inside a DC. You can specify a matrix of weights (distances) for the availability zones.

New zones are added by calling a corresponding API method of the orchestrator.

By default, the matrix of weights (distances) for the zones is not configured, and geo-redundancy for such configurations works as follows:

  • Data is always written to the master.
  • If the master is available, then it is used for reading.
  • If the master is unavailable, then any available replica is used for reading.

When you define a matrix of weights (distances) by calling /api/v1/zones/weights, the automatic scale-out system of the Tarantool DBMS finds a replica which is the closest to the specified router in terms of weights, and starts using this replica for reading. If this replica is not available, then the next nearest replica is selected, taking into account the distances specified in the configuration.

Appendix E. Orchestrator API reference

Configuring the zones

POST /api/v1/zone

Create a new zone.

Request

{
"name": "zone 1"
}

Response

{
"error": {
    "code": 0,
    "message": "ok"
},
"data": {
    "id": 2,
    "name": "zone 2"
},
"status": true
}

Potential errors

  • zone_exists - the specified zone already exists
GET /api/v1/zone/{zone_id: optional}

Return information on the specified zone or on all the zones.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": [
        {
            "id": 1,
            "name": "zone 11"
        },
        {
            "id": 2,
            "name": "zone 2"
        }
    ],
    "status": true
}

Potential errors

  • zone_not_found - the specified zone is not found
PUT /api/v1/zone/{zone_id}

Update information on the zone.

Body

{
    "name": "zone 22"
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • zone_not_found - the specified zone is not found
DELETE /api/v1/zone/{zone_id}

Delete a zone if it doesn’t store any nodes.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • zone_not_found - the specified zone is not found
  • zone_in_use - the specified zone stores at least one node

Configuring the zone weights

POST /api/v1/zones/weights

Set the zone weights configuration.

Body

{
    "weights": {
        "1": {
            "2": 10,
            "3": 11
        },
        "2": {
            "1": 10,
            "3": 12
        },
        "3": {
            "1": 11,
            "2": 12
        }
    }
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • zones_weights_error - configuration error
GET /api/v1/zones/weights

Return the zone weights configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {
        "1": {
            "2": 10,
            "3": 11
        },
        "2": {
            "1": 10,
            "3": 12
        },
        "3": {
            "1": 11,
            "2": 12
        }
    },
    "status": true
}

Potential errors

  • zone_not_found - the specified zone is not found

Configuring registry

GET /api/v1/registry/nodes/new

Return all the detected nodes.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": [
        {
            "uuid": "uuid-2",
            "hostname": "tnt2.public.i",
            "name": "tnt2"
        }
    ],
    "status": true
}
POST /api/v1/registry/node

Register the detected node.

Body

{
    "zone_id": 1,
    "uuid": "uuid-2",
    "uri": "tnt2.public.i:3301",
    "user": "user1:pass1",
    "repl_user": "repl_user1:repl_pass1",
    "cfg": {
        "listen": "0.0.0.0:3301"
    }
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • node_already_registered - the specified node is already registred
  • zone_not_found - the specified zone is not found
  • node_not_discovered - the specified node is not detected
PUT /api/v1/registry/node/{node_uuid}

Update the registered node parameters.

Body

Pass only those parameters that need to be updated.

{
    "zone_id": 1,
    "repl_user": "repl_user2:repl_pass2",
    "cfg": {
        "listen": "0.0.0.0:3301",
        "memtx_memory": 100000
    }
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • node_not_registered - the specified node is not registered
GET /api/v1/registry/node/{node_uuid: optional}

Return information on the nodes in a cluster. If node_uuid is passed, information on this node only is returned.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {
        "uuid-1": {
            "user": "user1:pass1",
            "hostname": "tnt1.public.i",
            "repl_user": "repl_user2:repl_pass2",
            "uri": "tnt1.public.i:3301",
            "zone_id": 1,
            "name": "tnt1",
            "cfg": {
                "listen": "0.0.0.0:3301",
                "memtx_memory": 100000
            },
            "zone": 1
        },
        "uuid-2": {
            "user": "user1:pass1",
            "hostname": "tnt2.public.i",
            "name": "tnt2",
            "uri": "tnt2.public.i:3301",
            "repl_user": "repl_user1:repl_pass1",
            "cfg": {
                "listen": "0.0.0.0:3301"
            },
            "zone": 1
        }
    },
    "status": true
}

Potential errors

  • node_not_registered - the specified node is not registered
DELETE /api/v1/registry/node/{node_uuid}

Delete the node if it doesn’t belong to any replica set.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • node_not_registered - the specified node is not registered
  • node_in_use - the specified node is in use by a replica set

Routers API

GET /api/v1/routers

Return the list of all nodes that constitute the router.

Response

{
    "data": [
        "uuid-1"
    ],
    "status": true,
    "error": {
        "code": 0,
        "message": "ok"
    }
}
POST /api/v1/routers

Assign the router role to the node.

Body

{
    "uuid": "uuid-1"
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • node_not_registered - the specified node is not registred
DELETE /api/v1/routers/{uuid}

Release the router role from the node.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Configuring replica sets

POST /api/v1/replicaset

Create a replica set containing all the registered nodes.

Body

{
    "uuid": "optional-uuid",
    "replicaset": [
        {
            "uuid": "uuid-1",
            "master": true
        }
    ]
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {
        "replicaset_uuid": "cc6568a2-63ca-413d-8e39-704b20adb7ae"
    },
    "status": true
}

Potential errors

  • replicaset_exists – the specified replica set already exists
  • replicaset_empty – the specified replica set doesn’t contain any nodes
  • node_not_registered – the specified node is not registered
  • node_in_use – the specified node is in use by another replica set
PUT /api/v1/replicaset/{replicaset_uuid}

Update the replica set parameters.

Body

{
    "replicaset": [
        {
            "uuid": "uuid-1",
            "master": true
        },
        {
            "uuid": "uuid-2",
            "master": false,
            "off": true
        }
    ]
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • replicaset_empty – the specified replica set doesn’t contain any nodes
  • replicaset_not_found – the specified replica set is not found
  • node_not_registered – the specified node is not registered
  • node_in_use – the specified node is in use by another replica set
GET /api/v1/replicaset/{replicaset_uuid: optional}

Return information on all the cluster components. If replicaset_uuid is passed, information on this replica set only is returned.

Body

{
    "name": "zone 22"
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {
        "cc6568a2-63ca-413d-8e39-704b20adb7ae": {
            "uuid-1": {
                "hostname": "tnt1.public.i",
                "off": false,
                "repl_user": "repl_user2:repl_pass2",
                "uri": "tnt1.public.i:3301",
                "master": true,
                "name": "tnt1",
                "user": "user1:pass1",
                "zone_id": 1,
                "zone": 1
            },
            "uuid-2": {
                "hostname": "tnt2.public.i",
                "off": true,
                "repl_user": "repl_user1:repl_pass1",
                "uri": "tnt2.public.i:3301",
                "master": false,
                "name": "tnt2",
                "user": "user1:pass1",
                "zone": 1
            }
        }
    },
    "status": true
}

Potential errors

  • replicaset_not_found – the specified replica set is not found
DELETE /api/v1/replicaset/{replicaset_uuid}

Delete a zone if it doesn’t store any nodes.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • replicaset_not_found - the specified replica set is not found
POST /api/v1/replicaset/{replicaset_uuid}/master

Switch the master in the replica set.

Body

{
    "instance_uuid": "uuid-1",
    "hostname_name": "hostname:instance_name"
}

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Potential errors

  • replicaset_not_found – the specified replica set is not found
  • node_not_registered – the specified node is not registered
  • node_not_in_replicaset – the specified node is not in the specified replica set
POST /api/v1/replicaset/{replicaset_uuid}/node

Add a node to the replica set.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {},
    "status": true
}

Body

{
    "instance_uuid": "uuid-1",
    "hostname_name": "hostname:instance_name",
    "master": false,
    "off": false
}

Potential errors

  • replicaset_not_found – the specified replica set is not found
  • node_not_registered – the specified node is not registered
  • node_in_use – the specified node is in use by another replica set
GET /api/v1/replicaset/status

Return statistics on the cluster.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "data": {
        "cluster": {
            "routers": [
                {
                    "zone": 1,
                    "name": "tnt1",
                    "repl_user": "repl_user1:repl_pass1",
                    "hostname": "tnt1.public.i",
                    "status": null,
                    "uri": "tnt1.public.i:3301",
                    "user": "user1:pass1",
                    "uuid": "uuid-1",
                    "total_rps": null
                }
            ],
            "storages": [
                {
                    "hostname": "tnt1.public.i",
                    "repl_user": "repl_user2:repl_pass2",
                    "uri": "tnt1.public.i:3301",
                    "name": "tnt1",
                    "total_rps": null,
                    "status": 'online',
                    "replicas": [
                        {
                            "user": "user1:pass1",
                            "hostname": "tnt2.public.i",
                            "replication_info": null,
                            "repl_user": "repl_user1:repl_pass1",
                            "uri": "tnt2.public.i:3301",
                            "uuid": "uuid-2",
                            "status": 'online',
                            "name": "tnt2",
                            "total_rps": null,
                            "zone": 1
                        }
                    ],
                    "user": "user1:pass1",
                    "zone_id": 1,
                    "uuid": "uuid-1",
                    "replicaset_uuid": "cc6568a2-63ca-413d-8e39-704b20adb7ae",
                    "zone": 1
                }
            ]
        }
    },
    "status": true
}

Potential errors

  • zone_not_found - the specified zone is not found
  • zone_in_use - the specified zone stores at least one node

Setting up configuration versions

POST /api/v1/version

Set the zone weights configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {
        "version": 2
    }
}

Potential errors

  • cfg_error - configuration error
GET /api/v1/version

Return the zone weights configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {
        "version": 2
    }
}

Configuring sharding

POST /api/v1/sharding/cfg

Add a new sharding configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {}
}
GET /api/v1/sharding/cfg

Return the current sharding configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {}
}

Resetting cluster configuration

POST /api/v1/clean/cfg

Reset the cluster configuration.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {}
}
POST /api/v1/clean/all

Reset the cluster configuration and delete information on the cluster nodes from the ZooKeeper catalogues.

Response

{
    "error": {
        "code": 0,
        "message": "ok"
    },
    "status": true,
    "data": {}
}