Data model

Lua versus MsgPack

Field Type Details

nil. In Lua, the nil type has only one possible value, also called nil. Tarantool displays it as null when using the default YAML format. Nil may be compared to values of any types with == (is-equal) or ~= (is-not-equal), but other comparison operations will not work. Nil may not be used in Lua tables; the workaround is to use box.NULL because nil == box.NULL is true. Example: nil.

boolean. A boolean is either true or false. Example: true.

integer. The Tarantool integer type is for integers between -9223372036854775808 and 18446744073709551615, which is about 18 quintillion. This type corresponds to the number type in Lua and to the integer type in MsgPack. Example: -2^63.

unsigned. The Tarantool unsigned type is for integers between 0 and 18446744073709551615. So it is a subset of integer. Example: 123456.

double. The double field type exists mainly to be equivalent to Tarantool/SQL’s DOUBLE data type. In msgpuck.h (Tarantool’s interface to MsgPack), the storage type is MP_DOUBLE and the size of the encoded value is always 9 bytes. In Lua, fields of the double type can only contain non-integer numeric values and cdata values with double floating-point numbers. Examples: 1.234, -44, 1.447e+44.

To avoid using the wrong kind of values inadvertently, use ffi.cast() when searching or changing double fields. For example, instead of space_object:insert{value} use ffi = require('ffi') ... space_object:insert({ffi.cast('double',value)}). Example:

s = box.schema.space.create('s', {format = {{'d', 'double'}}})
s:create_index('ii')
s:insert({1.1})
ffi = require('ffi')
s:insert({ffi.cast('double', 1)})
s:insert({ffi.cast('double', tonumber('123'))})
s:select(1.1)
s:select({ffi.cast('double', 1)})

Arithmetic with cdata double will not work reliably, so for Lua, it is better to use the number type. This warning does not apply for Tarantool/SQL because Tarantool/SQL does implicit casting.

number. The Tarantool number field may have both integer and floating-point values, although in Lua a number is a double-precision floating-point.

Tarantool will try to store a Lua number as floating-point if the value contains a decimal point or is very large (greater than 100 trillion = 1e14), otherwise Tarantool will store it as an integer. To ensure that even very large numbers are stored as integers, use the tonumber64 function, or the LL (Long Long) suffix, or the ULL (Unsigned Long Long) suffix. Here are examples of numbers using regular notation, exponential notation, the ULL suffix and the tonumber64 function: -55, -2.7e+20, 100000000000000ULL, tonumber64('18446744073709551615').

decimal. The Tarantool decimal type is stored as a MsgPack ext (Extension). Values with the decimal type are not floating-point values although they may contain decimal points. They are exact with up to 38 digits of precision. Example: a value returned by a function in the decimal module.

string. A string is a variable-length sequence of bytes, usually represented with alphanumeric characters inside single quotes. In both Lua and MsgPack, strings are treated as binary data, with no attempts to determine a string’s character set or to perform any string conversion – unless there is an optional collation. So, usually, string sorting and comparison are done byte-by-byte, without any special collation rules applied. For example, numbers are ordered by their point on the number line, so 2345 is greater than 500; meanwhile, strings are ordered by the encoding of the first byte, then the encoding of the second byte, and so on, so '2345' is less than '500'. Example: 'A, B, C'.

bin. A bin (binary) value is not directly supported by Lua but there is a Tarantool type varbinary which is encoded as MsgPack binary. For an (advanced) example showing how to insert varbinary into a database, see the Cookbook Recipe for ffi_varbinary_insert. Example: "\65 \66 \67".

uuid. The Tarantool uuid type is used for Universally Unique Identifiers. Since version 2.4.1 Tarantool stores uuid values as a MsgPack ext (Extension).

Example: 64d22e4d-ac92-4a23-899a-e5934af5479.

array. An array is represented in Lua with {...} (braces). Examples: lists of numbers representing points in geometric figures: {10, 11}, {3, 5, 9, 10}.

table. Lua tables with string keys are stored as MsgPack maps; Lua tables with integer keys starting with 1 are stored as MsgPack arrays. Nils may not be used in Lua tables; the workaround is to use box.NULL. Example: a box.space.tester:select() request will return a Lua table.

tuple. A tuple is a light reference to a MsgPack array stored in the database. It is a special type (cdata) to avoid conversion to a Lua table on retrieval. A few functions may return tables with multiple tuples. For tuple examples, see box.tuple.

scalar. Values in a scalar field can be boolean, integer, unsigned, double, number, decimal, string, uuid, or varbinary; but not array, map, or tuple. Examples: true, 1, 'xxx'.

any. Values in a field of this type can be boolean, integer, unsigned, double, number, decimal, string, uuid, varbinary, array, map, or tuple. Examples: true, 1, 'xxx', {box.NULL, 0}.

Examples of insert requests with different field types:

tarantool> box.space.K:insert{1,nil,true,'A B C',12345,1.2345}
---
- [1, null, true, 'A B C', 12345, 1.2345]
...
tarantool> box.space.K:insert{2,{['a']=5,['b']=6}}
---
- [2, {'a': 5, 'b': 6}]
...
tarantool> box.space.K:insert{3,{1,2,3,4,5}}
---
- [3, [1, 2, 3, 4, 5]]
...

Indexed field types

Indexes restrict values that Tarantool can store with MsgPack. This is why, for example, 'unsigned' and 'integer' are different field types, although in MsgPack they are both stored as integer values. An 'unsigned' index contains only non-negative integer values, while an ‘integer’ index contains any integer values.

Here again are the field types described in Field Type Details, and the index types they can fit in. The default field type is 'unsigned' and the default index type is TREE. Although 'nil' is not a legal indexed field type, indexes may contain nil as a non-default option. Full information is in section Details about index field types.

Options for box.schema.sequence.create()

Once a sequence exists, it can be altered, dropped, reset, forced to generate the next value, or associated with an index.

For an initial example, we generate a sequence named ‘S’.

tarantool> box.schema.sequence.create('S',{min=5, start=5})
---
- step: 1
  id: 5
  min: 5
  cache: 0
  uid: 1
  max: 9223372036854775807
  cycle: false
  name: S
  start: 5
...

The result shows that the new sequence has all default values, except for the two that were specified, min and start.

Then we get the next value, with the next() function.

tarantool> box.sequence.S:next()
---
- 5
...

The result is the same as the start value. If we called next() again, we would get 6 (because the previous value plus the step value is 6), and so on.

Then we create a new table and specify that its primary key should be generated from the sequence.

tarantool> s=box.schema.space.create('T')
---
...
tarantool> s:create_index('I',{sequence='S'})
---
- parts:
  - type: unsigned
    is_nullable: false
    fieldno: 1
  sequence_id: 1
  id: 0
  space_id: 520
  unique: true
  type: TREE
  sequence_fieldno: 1
  name: I
...
---
...

Then we insert a tuple without specifying a value for the primary key.

tarantool> box.space.T:insert{nil,'other stuff'}
---
- [6, 'other stuff']
...

The result is a new tuple where the first field has a value of 6. This arrangement, where the system automatically generates the values for a primary key, is sometimes called “auto-incrementing” or “identity”.

For syntax and implementation details, see the reference for box.schema.sequence.

Data operations

The basic data operations supported in Tarantool are:

• five data-manipulation operations (INSERT, UPDATE, UPSERT, DELETE, REPLACE), and
• one data-retrieval operation (SELECT).

All of them are implemented as functions in box.space submodule.

Examples:

• INSERT: Add a new tuple to space ‘tester’.

  The first field, field[1], will be 999 (MsgPack type is integer).

  The second field, field[2], will be ‘Taranto’ (MsgPack type is string).

  tarantool> box.space.tester:insert{999, 'Taranto'}
  

• UPDATE: Update the tuple, changing field field[2].

  The clause “{999}”, which has the value to look up in the index of the tuple’s primary-key field, is mandatory, because update() requests must always have a clause that specifies a unique key, which in this case is field[1].

  The clause “{{‘=’, 2, ‘Tarantino’}}” specifies that assignment will happen to field[2] with the new value.

  tarantool> box.space.tester:update({999}, {{'=', 2, 'Tarantino'}})
  

• UPSERT: Upsert the tuple, changing field field[2] again.

  The syntax of upsert() is similar to the syntax of update(). However, the execution logic of these two requests is different. UPSERT is either UPDATE or INSERT, depending on the database’s state. Also, UPSERT execution is postponed until after transaction commit, so, unlike update(), upsert() doesn’t return data back.

  tarantool> box.space.tester:upsert({999, 'Taranted'}, {{'=', 2, 'Tarantism'}})
  

• REPLACE: Replace the tuple, adding a new field.

  This is also possible with the update() request, but the update() request is usually more complicated.

  tarantool> box.space.tester:replace{999, 'Tarantella', 'Tarantula'}
  

• SELECT: Retrieve the tuple.

  The clause “{999}” is still mandatory, although it does not have to mention the primary key.

  tarantool> box.space.tester:select{999}
  

• DELETE: Delete the tuple.

  In this example, we identify the primary-key field.

  tarantool> box.space.tester:delete{999}
  

Summarizing the examples:

• Functions insert and replace accept a tuple (where a primary key comes as part of the tuple).
• Function upsert accepts a tuple (where a primary key comes as part of the tuple), and also the update operations to execute.
• Function delete accepts a full key of any unique index (primary or secondary).
• Function update accepts a full key of any unique index (primary or secondary), and also the operations to execute.
• Function select accepts any key: primary/secondary, unique/non-unique, full/partial.

See reference on box.space for more details on using data operations.

Complexity factors

In reference for box.space and Submodule box.index submodules, there are notes about which complexity factors might affect the resource usage of each function.

Data schema description in a code file

The code file is usually called init.lua and contains the following schema description:

box.cfg()

users = box.schema.create_space('users', { if_not_exists = true })
users:format({{ name = 'user_id', type = 'number'}, { name = 'fullname', type = 'string'}})

users:create_index('pk', { parts = { { field = 'user_id', type = 'number'}}})

This is quite simple: when you run tarantool, it executes this code and creates a data schema. To run this file, use:

tarantool init.lua

However, it may seem complicated if you do not plan to dive deep into the Lua language and its syntax.

Possible difficulty: the snippet above has a function call with a colon: users:format. It is used to pass the users variable as the first argument of the format function. This is similar to self in object-based languages.

So it might be more convenient for you to describe the data schema with YAML.

Data schema description using the DDL module

The DDL module allows you to describe a data schema in the YAML format in a declarative way.

The schema would look something like this:

spaces:
    users:
      engine: memtx
      is_local: false
      temporary: false
      format:
      - {name: user_id, type: uuid, is_nullable: false}
      - {name: fullname, type: string,  is_nullable: false}
      indexes:
      - name: user_id
        unique: true
        parts: [{path: user_id, type: uuid, is_nullable: false}]
        type: HASH

This alternative is simpler to use, and you do not have to dive deep into Lua.

DDL is a built-in Cartridge module. Cartridge is a cluster solution for Tarantool. In its WebUI, there is a separate tab called “Schema”. On this tab, you can define the schema, check its correctness, and apply it to the whole cluster.

If you do not use Cartridge, you can still use the DDL module: put the following Lua code into the file that you use to run Tarantool. This file is usually called init.lua.

local yaml = require('yaml')
local ddl = require('ddl')

box.cfg{}

local fh = io.open('ddl.yml', 'r')
local schema = yaml.decode(fh:read('*all'))
fh:close()
local ok, err = ddl.check_schema(schema)
if not ok then
    print(err)
end
local ok, err = ddl.set_schema(schema)
if not ok then
    print(err)
end

Adding a field to the end of a space

You can add a field as follows:

local users = box.space.users
local fmt = users:format()

table.insert(fmt, { name = 'age', type = 'number', is_nullable = true })
users:format(fmt)

Note that the field must have the is_nullable parameter. Otherwise, an error will occur.

After creating a new field, you probably want to fill it with data. The tarantool/moonwalker module is useful for this task. The README file describes how to work with this module.

Creating an index

Index creation is described in the space_object:create_index() method.

Other types of migrations

Other types of migrations are also allowed, but it would be more difficult to maintain data consistency.

Migrations are possible in two cases:

• When Tarantool starts, and no client uses the database yet
• During request processing, when active clients are already using the database

For the first case, it is enough to write and test the migration code. The most difficult task is to migrate data when there are active clients. You should keep it in mind when you initially design the data schema.

We identify the following problems if there are active clients:

• Associated data can change atomically.
• The system should be able to transfer data using both the new schema and the old one.
• When data is being transferred to a new space, data access should consider that the data might be in one space or another.
• Write requests must not interfere with the migration. A common approach is to write according to the new data schema.

These issues may or may not be relevant depending on your application and its availability requirements.

What you need to know when writing complex migrations

Tarantool has a transaction mechanism. It is useful when writing a migration, because it allows you to work with the data atomically. But before using the transaction mechanism, you should explore its limitations.

For details, see the section about transactions.

How you can apply migration

The migration code is executed on a running Tarantool instance. Important: no method guarantees you transactional application of migrations on the whole cluster.

Method 1: include migrations in the application code

This is quite simple: when you reload the code, the data is migrated at the right moment, and the database schema is updated. However, this method may not work for everyone. You may not be able to restart Tarantool or update the code using the hot-reload mechanism.

Method 2: tarantool/migrations (only for a Tarantool Cartridge cluster)

This method is described in the README file of the tarantool/migrations module.

$ tarantoolctl connect admin:password@localhost:3301