Updated at 2023-06-07 03:30:05.409502
Overview
Tarantool comes in two editions: the open-source Community Edition (CE)
and the commercial Enterprise Edition (EE).
Tarantool CE lets you develop applications and speed up a system in operation.
It features synchronous replication, affords easy scalability,
and includes tools to develop efficient applications.
The Tarantool community helps with any practical questions
regarding the Community Edition.
Tarantool EE provides advanced tools for
administration, deployment, and security management, along with premium support services.
This edition includes all the Community Edition features
and is more predictable in terms of solution cost and maintenance.
The Enterprise Edition is shipped as an SDK and includes a number of closed-source modules.
See the documentation for Tarantool EE.
- Primary storage
- No secondary storage required
- Tolerance to high write loads
- Support of relational approaches
- Composite secondary indexes
- Predictable request latency
- Write-behind caching
- Secondary index support
- Complex invalidation algorithm support
- Support of various identification techniques
- Advanced task lifecycle management
- Task scheduling
- Archiving of completed tasks
Data-centric applications
- Arbitrary data flows from many sources
- Incoming data processing
- Storage
- Background cycle processing
Getting started
The First steps section
will get you acquainted with Tarantool in 15 minutes.
We will be creating a basic microservice for TikTok.
We will start Tarantool, create a data schema, and write our first data.
You’ll get an understanding of the technology and learn about the basic terms and features.
In the Connecting to cluster section,
we’ll show you how to read or write data to Tarantool
from your Python/Go/PHP application or another programming language.
After connecting to the database for the first time, you might want to change the data schema.
In the section Updating the data schema,
we’ll discuss the approaches to changing the data schema and the associated limitations.
To make our code work with Tarantool,
we may want to transfer some of our data logic to Tarantool.
In the section Writing cluster application code,
we’ll write a “Hello, World!” program in the Lua language,
which will work in our Tarantool cluster.
This will give you a basic understanding of how the role mechanism works.
In this way, you’ll understand what part of your business logic you would like
to write in/migrate to Tarantool.
To continue exploring Tarantool and its ecosystem, you might want to check out
Tarantool tutorials and guides.
The Cartridge beginner tutorial can also be found there.
First steps
This is the recommended guide for getting to know the product.
Note
You also might want to check out our
basic Tarantool tutorial.
It shows how to launch one Tarantool instance,
create a space, build an index, and write data.
We recommend that beginners go through the current tutorial first
and then see the basic tutorial to dive deeper into the product.
If you just want to run the complete tutorial code quickly, go to
Launching an application.
Launch in the cloud
This tutorial is also available in the cloud. It’s free, and it’s the fastest way to start.
To follow this tutorial in the cloud, go to try.tarantool.io.
However, you will still need to install Tarantool
if you want to get better acquainted with it.
Run locally
For Linux/macOS users:
Install Tarantool from the Download page.
Install Node.js, which is required for the tutorial frontend.
Install the cartridge-cli utility through your package manager:
sudo yum install cartridge-cli
brew install cartridge-cli
To learn more, check the cartridge-cli
installation guide.
Clone the Getting Started tutorial repository.
Everything is ready and organized in this repository.
In the cloned directory, run the following:
cartridge build
cartridge start
Note
In case of a problem with cartridge build, run it with the --verbose flag
to learn about the source of the problem. If there is a problem with Node.js (npm):
- Check that Node.js is in the
$PATH.
- Try forcefully removing the
node_modules directory from the dependencies’ directories:
rm -rf analytics/node_modules front/node_modules
After that, try running cartridge build again.
If all else fails, please file us an issue on GitHub.
You’re all set! At http://localhost:8081, you will see the Tarantool Cartridge UI.
Running in Docker:
docker run -p 3301:3301 -p 8081:8081 tarantool/getting-started
That’s it! At http://localhost:8081, you will see the Tarantool Cartridge UI.
For Windows users:
Use Docker to get started.
Configuring a cluster [1 minute]
Everything you need to know to get started:
A Tarantool cluster has two service roles: router and storage.
- Storage is used to store data.
- Router is an intermediary between clients and storages.
It accepts a client’s request, takes data from the proper storage,
and returns it to the client.
We see that we have 5 unconfigured instances on the “Cluster” tab.
Let’s create one router and one storage for a start.
First, click the “Configure” button on the “router” instance and configure
it as in the screenshot below:
Next, we configure the “s1-master” instance:
It will look something like this:
Let’s enable sharding in the cluster using the “Bootstrap vshard” button. It is
located in the top right corner.
Creating a data schema [2 minutes]
Let’s start with the data schema – take a look at the Code tab on the left.
There you can find a file called schema.yml. In this file, you can
describe the entire cluster’s data schema, edit the current schema,
validate its correctness, and apply it to the whole cluster.
First, let’s create the necessary tables. In Tarantool, they are called spaces.
We need to store:
- Users
- Videos with descriptions
- Likes for each video
Copy the schema description from the code block below and paste it in the schema.yml file on the Code tab.
Click the “Apply” button.
After that, the data schema will be described in the cluster.
This is what our data schema will look like:
spaces:
users:
engine: memtx
is_local: false
temporary: false
sharding_key:
- "user_id"
format:
- {name: bucket_id, type: unsigned, is_nullable: false}
- {name: user_id, type: uuid, is_nullable: false}
- {name: fullname, type: string, is_nullable: false}
indexes:
- name: user_id
unique: true
parts: [{path: user_id, type: uuid, is_nullable: false}]
type: HASH
- name: bucket_id
unique: false
parts: [{path: bucket_id, type: unsigned, is_nullable: false}]
type: TREE
videos:
engine: memtx
is_local: false
temporary: false
sharding_key:
- "video_id"
format:
- {name: bucket_id, type: unsigned, is_nullable: false}
- {name: video_id, type: uuid, is_nullable: false}
- {name: description, type: string, is_nullable: true}
indexes:
- name: video_id
unique: true
parts: [{path: video_id, type: uuid, is_nullable: false}]
type: HASH
- name: bucket_id
unique: false
parts: [{path: bucket_id, type: unsigned, is_nullable: false}]
type: TREE
likes:
engine: memtx
is_local: false
temporary: false
sharding_key:
- "video_id"
format:
- {name: bucket_id, type: unsigned, is_nullable: false}
- {name: like_id, type: uuid, is_nullable: false}
- {name: user_id, type: uuid, is_nullable: false}
- {name: video_id, type: uuid, is_nullable: false}
- {name: timestamp, type: string, is_nullable: true}
indexes:
- name: like_id
unique: true
parts: [{path: like_id, type: uuid, is_nullable: false}]
type: HASH
- name: bucket_id
unique: false
parts: [{path: bucket_id, type: unsigned, is_nullable: false}]
type: TREE
It’s simple. Let’s take a closer look at the essential points.
Tarantool has two built-in storage engines: memtx and vinyl.
memtx stores all data in RAM while asynchronously writing to
disk so that nothing gets lost.
Vinyl is a classic engine for storing data on the
hard drive. It is optimized for write-intensive scenarios.
In TikTok, there are a lot of simultaneous readings and
posts: users watch videos, like them, and comment on them.
Therefore, let’s use memtx.
The configuration above describes three memtx spaces (tables)
and the necessary indexes for each of the spaces.
Each space has two indexes:
- The primary key, which is required to read/write data.
- An index on the bucket_id field, which is a service field used for sharding.
Important: The name bucket_id is reserved. If you choose
another name, sharding won’t work for this space.
If you don’t use sharding in your project, you can remove the second index.
To understand which field to shard data by, Tarantool uses
sharding_key. sharding_key points to fields in the space by
which database records will be sharded. There can be more than one such field, but
in this example, we will only use one. When some data is inserted,
Tarantool forms a hash from this field, calculates the bucket number,
and selects the storage to record the data into.
Yes, buckets can repeat, and each storage stores a specific range of buckets.
Here are a couple more interesting facts:
- The
parts field in the index description can contain several fields,
which allows building a composite index. You won’t need it in this tutorial.
- Tarantool does not support foreign keys, so you have to check manually
upon insertion
that
video_id and user_id exist in the likes space.
We will write data to the Tarantool cluster using the CRUD module.
You don’t have to specify the shard you want to read from or write to – the module
does it for you.
Important: All cluster operations must be performed only on the router
and using the CRUD module.
Let’s connect the CRUD module in the code and write three procedures:
- User creation
- Adding a video
- Liking a video
The procedures must be described in a special file. To do this, go to
the “Code” tab. Create a new directory called extensions, and
in this directory, create the file api.lua.
Paste the code below into api.lua and click “Apply”.
local cartridge = require('cartridge')
local crud = require('crud')
local uuid = require('uuid')
local json = require('json')
function add_user(request)
local fullname = request:post_param("fullname")
local result, err = crud.insert_object('users', {user_id = uuid.new(), fullname = fullname})
if err ~= nil then
return {body = json.encode({status = "Error!", error = err}), status = 500}
end
return {body = json.encode({status = "Success!", result = result}), status = 200}
end
function add_video(request)
local description = request:post_param("description")
local result, err = crud.insert_object('videos', {video_id = uuid.new(), description = description})
if err ~= nil then
return {body = json.encode({status = "Error!", error = err}), status = 500}
end
return {body = json.encode({status = "Success!", result = result}), status = 200}
end
function like_video(request)
local video_id = request:post_param("video_id")
local user_id = request:post_param("user_id")
local result, err = crud.insert_object('likes', {like_id = uuid.new(),
video_id = uuid.fromstr(video_id),
user_id = uuid.fromstr(user_id)})
if err ~= nil then
return {body = json.encode({status = "Error!", error = err}), status = 500}
end
return {body = json.encode({status = "Success!", result = result}), status = 200}
end
return {
add_user = add_user,
add_video = add_video,
like_video = like_video,
}
Setting up HTTP API [2 minutes]
Clients will visit the Tarantool cluster using the HTTP protocol.
The cluster already has a built-in HTTP server.
To configure HTTP paths, you need to write a configuration
file. Go to the “Code” tab. Create the file config.yml
in the extensions directory, which you created on the last step.
Paste the configuration example below into config.yml and click “Apply”.
---
functions:
add_user:
module: extensions.api
handler: add_user
events:
- http: {path: "/add_user", method: POST}
add_video:
module: extensions.api
handler: add_video
events:
- http: {path: "/add_video", method: POST}
like_video:
module: extensions.api
handler: like_video
events:
- http: {path: "/like_video", method: POST}
...
Done! Let’s make test requests from the console.
curl -X POST --data "fullname=Taran Tool" url/add_user
Note
In the requests, substitute url with the address of your sandbox.
The protocol must be strictly HTTP.
For example, if you’re following this tutorial with Try Tarantool, this request will look something like this
(note that your hash is different):
curl -X POST --data "fullname=Taran Tool" http://artpjcvnmwctc4qppejgf57.try.tarantool.io/add_user
But if you’ve bootstrapped Tarantool locally, the request will look as follows:
curl -X POST --data "fullname=Taran Tool" http://localhost:8081/add_user
We’ve just created a user and got their UUID. Let’s remember it.
curl -X POST --data "description=My first tiktok" url/add_video
Let’s say a user has added their first video with a description.
The video clip also has a UUID. Let’s remember it, too.
In order to “like” the video, you need to specify the user UUID and the video UUID from the previous steps.
Substitute the ellipses in the command below with the corresponding UUIDs:
curl -X POST --data "video_id=...&user_id=..." url/like_video
The result will be something like this:
In our example, you can “like” the video as many times as you want.
It makes no sense in the real life, but it will help us understand how
sharding works – more precisely, the sharding_key parameter.
Our sharding_key for the likes is video_id.
We also specified a sharding_key for the videos space. It means
that likes will be stored on the same storage as videos.
This ensures data locality with regard to storage and allows
getting all the information you need in one network trip to Storage.
More details are described on the next step.
Looking at the data [1 minute]
Note
The following instructions are for Tarantool Enterprise Edition and the Try Tarantool cloud service.
The Space-Explorer tool is unavailable in the open-source version. Use the console to view data.
Check our documentation to learn more about data viewing.
To learn how to connect to a Tarantool instance, read the basic Tarantool manual.
Go to the “Space-Explorer” tab to see all the nodes in the cluster.
As we have only one storage and one router started so far, the data is stored
on only one node.
Let’s go to the node s1-master: click “Connect” and select the necessary space.
Check that everything is in place and move on.
Scaling the cluster [1 minute]
Let’s create a second shard. Click on the “Cluster” tab, select
s2-master, and click “Configure”. Select the roles as shown in the picture:
Click on the necessary roles and create a shard (replica set).
Checking how sharding works [1 minute]
Now we have two shards – two logical nodes that
share data among themselves. The router decides what piece of data goes to what shard.
By default, the router uses the hash function from the field sharding_key
we’ve specified in the DDL.
To enable a new shard, you need to set its weight to one.
Go back to the “Cluster” tab, open the s2-master settings,
set the Replica set weight to 1, and apply.
Something has already happened. Let’s go to Space-Explorer and check the node
s2-master. It turns out that some of the data from the first shard
has already migrated here! The scaling is done automatically.
Now let’s try adding more data to the cluster via the HTTP API.
We can check back later and make sure that the new data is also evenly
distributed across the two shards.
Disconnecting a shard for a while [1 minute]
In the s1-master settings, set Replica set weight to 0 and
apply. Wait for a few seconds, then go to Space-Explorer and look at the
data in s2-master. You will see that all the data has been migrated to
the remaining shard automatically.
Now we can safely disable the first shard for maintenance.
- README of the DDL module to create
your own data schema.
- README of the CRUD module to
learn more about the API and create your own cluster queries.
To continue to the next steps of the tutorial,
click the button in the bottom right corner
or select the section in the table of contents on the left.
Connecting to the cluster
In the last section, we set up a cluster, created a schema, and wrote data through the HTTP API.
Now we can connect to the cluster from code and work with data.
You may have noticed that we used the crud module in the HTTP handler code.
The code looked something like this:
local crud = require ('crud')
function add_user(request)
local result, err = crud.insert_object ('users', {user_id = uuid.new (), fullname = fullname})
end
This module allows you to work with data in a cluster. The syntax here is similar to
what the Tarantool box module offers.
You will learn more about the box module in the following sections.
The crud module contains a set of stored procedures.
To work with them, we must activate special roles on all routers and storages.
We selected those roles in the previous section, so we don’t need to do anything.
The roles are named accordingly: “crud-router” and “crud-storage”.
To write and read data in the Tarantool cluster from code, we will call stored
procedures of the crud module.
In Python, it looks like this:
res = conn.call('crud.insert', 'users', <uuid>, 'Jim Carrey')
users = conn.call('crud.select', 'users', {limit: 100})
All functions of the crud module are described
in the README of our GitHub repository.
Here is an incomplete list:
insertselectgetdeletemin/maxreplace/upserttruncate
To learn how to call stored procedures in your programming language, see the corresponding section:
For connectors to other languages, check the README for the connector of your choice
on GitHub.
Updating the data schema
When working with data, it is sometimes necessary to change the original data schema.
In the previous sections, we described a cluster-wide data schema in the YAML format.
The ddl module is responsible for applying the schema on the cluster. This module does not allow
to modify the schema after applying it.
The easiest way to change it is to delete the database snapshots and create
a schema from scratch. Of course, this is only acceptable during application
development and debugging.
For production scenarios, read the section on migrations.
To remove snapshots:
- If you are using Tarantool in the cloud,
click the “Reset configuration” button.
- If you’ve started Tarantool locally via
cartridge start,
run cartridge clean in the application directory.
- If you’ve started Tarantool in a different way,
delete the snapshots and xlogs manually.
These files have the .snap and .xlog extensions respectively,
and they are located in the Tarantool working directory.
To understand how the Tarantool data schema works, read the Data model section.
Writing code in a cluster application
In the “Getting Started” tutorial,
we wrote the application code directly in the browser.
We used the file config.yml to describe HTTP endpoint handlers.
This is a convenient and fast way to write code
that allows you to use Tarantool as a repository without any additional HTTP service.
This functionality is implemented through the cartridge-extensions module.
It is also included in the tutorial default application.
However, in Tarantool, you can implement absolutely any business logic on top of a cluster.
This Cartridge getting started section
covers the cluster roles mechanism and writing a cluster application from scratch.
How-to guides
This chapter contains practical examples as well as
tutorials for those who would like to dig deeper into Tarantool usage.
If you are new to Tarantool, please see our
Getting Started guides first.
Connecting from your favorite language
In the previous sections,
you have learned how to create a Tarantool database.
Now let’s see how to connect to the database from different programming
languages, such as Python, PHP, Go, and C++, and
execute typical requests for manipulating the data (select, insert, delete,
and so on).
Connecting from Python
Before we proceed:
Install
the tarantool module. We recommend using python3 and pip3.
Start Tarantool (locally or in Docker)
and make sure that you have created and populated a database as we suggested
earlier:
box.cfg{listen = 3301}
s = box.schema.space.create('tester')
s:format({
{name = 'id', type = 'unsigned'},
{name = 'band_name', type = 'string'},
{name = 'year', type = 'unsigned'}
})
s:create_index('primary', {
type = 'hash',
parts = {'id'}
})
s:create_index('secondary', {
type = 'hash',
parts = {'band_name'}
})
s:insert{1, 'Roxette', 1986}
s:insert{2, 'Scorpions', 2015}
s:insert{3, 'Ace of Base', 1993}
Important
Please do not close the terminal window
where Tarantool is running – you’ll need it soon.
In order to connect to Tarantool as an administrator, reset the password
for the admin user:
box.schema.user.passwd('pass')
A space is a container for tuples.
To access a space as a named object, use connection.space:
>>> tester = connection.space('tester')
To insert a tuple into a space, use insert:
>>> tester.insert((4, 'ABBA', 1972))
[4, 'ABBA', 1972]
Let’s start with selecting a tuple by the primary key
(in our example, this is the index named primary, based on the id field
of each tuple). Use select:
>>> tester.select(4)
[4, 'ABBA', 1972]
Next, select tuples by a secondary key.
For this purpose, you need to specify the number or name of the index.
First off, select tuples using the index number:
>>> tester.select('Scorpions', index=1)
[2, 'Scorpions', 2015]
(We say index=1 because index numbers in Tarantool start with 0,
and we’re using our second index here.)
Now make a similar query by the index name and make sure that the result
is the same:
>>> tester.select('Scorpions', index='secondary')
[2, 'Scorpions', 2015]
Finally, select all the tuples in a space via a select with no
arguments:
Update a field value using update:
>>> tester.update(4, [('=', 1, 'New group'), ('+', 2, 2)])
This updates the value of field 1 and increases the value of field 2
in the tuple with id = 4. If a tuple with this id doesn’t exist,
Tarantool will return an error.
Now use replace to totally replace the tuple that matches the
primary key. If a tuple with this primary key doesn’t exist, Tarantool will
do nothing.
>>> tester.replace((4, 'New band', 2015))
You can also update the data using upsert that works similarly
to update, but creates a new tuple if the old one was not found.
>>> tester.upsert((4, 'Another band', 2000), [('+', 2, 5)])
This increases by 5 the value of field 2 in the tuple with id = 4, or
inserts the tuple (4, "Another band", 2000) if a tuple with this id
doesn’t exist.
To delete a tuple, use delete(primary_key):
>>> tester.delete(4)
[4, 'New group', 2012]
To delete all tuples in a space (or to delete an entire space), use call.
We’ll focus on this function in more detail in the
next section.
To delete all tuples in a space, call space:truncate:
>>> connection.call('box.space.tester:truncate', ())
To delete an entire space, call space:drop.
This requires connecting to Tarantool as the admin user:
>>> connection.call('box.space.tester:drop', ())
Executing stored procedures
Switch to the terminal window where Tarantool is running.
Note
If you don’t have a terminal window with remote connection to Tarantool,
check out these guides:
Define a simple Lua function:
function sum(a, b)
return a + b
end
Now we have a Lua function defined in Tarantool. To invoke this function from
python, use call:
>>> connection.call('sum', (3, 2))
5
To send bare Lua code for execution, use eval:
>>> connection.eval('return 4 + 5')
9
Connecting from PHP
Before we proceed:
Install
the tarantool/client library.
Start Tarantool (locally or in Docker)
and make sure that you have created and populated a database as we suggested
earlier:
box.cfg{listen = 3301}
s = box.schema.space.create('tester')
s:format({
{name = 'id', type = 'unsigned'},
{name = 'band_name', type = 'string'},
{name = 'year', type = 'unsigned'}
})
s:create_index('primary', {
type = 'hash',
parts = {'id'}
})
s:create_index('secondary', {
type = 'hash',
parts = {'band_name'}
})
s:insert{1, 'Roxette', 1986}
s:insert{2, 'Scorpions', 2015}
s:insert{3, 'Ace of Base', 1993}
Important
Please do not close the terminal window
where Tarantool is running – you’ll need it soon.
In order to connect to Tarantool as an administrator, reset the password
for the admin user:
box.schema.user.passwd('pass')
A space is a container for tuples. To access a space as a named object,
use getSpace:
$tester = $client->getSpace('tester');
To insert a tuple into a space, use insert:
$result = $tester->insert([4, 'ABBA', 1972]);
Let’s start with selecting a tuple by the primary key
(in our example, this is the index named primary, based on the id field
of each tuple). Use select:
use Tarantool\Client\Schema\Criteria;
$result = $tester->select(Criteria::key([4]));
printf(json_encode($result));
Next, select tuples by a secondary key.
For this purpose, you need to specify the number or name of the index.
First off, select tuples using the index number:
$result = $tester->select(Criteria::index(1)->andKey(['Scorpions']));
printf(json_encode($result));
(We say index(1) because index numbers in Tarantool start with 0,
and we’re using our second index here.)
Now make a similar query by the index name and make sure that the result
is the same:
$result = $tester->select(Criteria::index('secondary')->andKey(['Scorpions']));
printf(json_encode($result));
Finally, select all the tuples in a space via a select:
$result = $tester->select(Criteria::allIterator());
Update a field value using update:
use Tarantool\Client\Schema\Operations;
$result = $tester->update([4], Operations::set(1, 'New group')->andAdd(2, 2));
This updates the value of field 1 and increases the value of field 2
in the tuple with id = 4. If a tuple with this id doesn’t exist,
Tarantool will return an error.
Now use replace to totally replace the tuple that matches the
primary key. If a tuple with this primary key doesn’t exist, Tarantool will
do nothing.
$result = $tester->replace([4, 'New band', 2015]);
You can also update the data using upsert that works similarly
to update, but creates a new tuple if the old one was not found.
use Tarantool\Client\Schema\Operations;
$tester->upsert([4, 'Another band', 2000], Operations::add(2, 5));
This increases by 5 the value of field 2 in the tuple with id = 4, or
inserts the tuple (4, "Another band", 2000) if a tuple with this id
doesn’t exist.
To delete a tuple, use delete(primary_key):
$result = $tester->delete([4]);
To delete all tuples in a space (or to delete an entire space), use call.
We’ll focus on this function in more detail in the
next section.
To delete all tuples in a space, call space:truncate:
$result = $client->call('box.space.tester:truncate');
To delete an entire space, call space:drop.
This requires connecting to Tarantool as the admin user:
$result = $client->call('box.space.tester:drop');
Executing stored procedures
Switch to the terminal window where Tarantool is running.
Note
If you don’t have a terminal window with remote connection to Tarantool,
check out these guides:
Define a simple Lua function:
function sum(a, b)
return a + b
end
Now we have a Lua function defined in Tarantool. To invoke this function from
php, use call:
$result = $client->call('sum', 3, 2);
To send bare Lua code for execution, use eval:
$result = $client->evaluate('return 4 + 5');
Connecting from Go
Before we proceed:
Install
the go-tarantool library.
Start Tarantool (locally or in Docker)
and make sure that you have created and populated a database as we suggested
earlier:
box.cfg{listen = 3301}
s = box.schema.space.create('tester')
s:format({
{name = 'id', type = 'unsigned'},
{name = 'band_name', type = 'string'},
{name = 'year', type = 'unsigned'}
})
s:create_index('primary', {
type = 'hash',
parts = {'id'}
})
s:create_index('secondary', {
type = 'hash',
parts = {'band_name'}
})
s:insert{1, 'Roxette', 1986}
s:insert{2, 'Scorpions', 2015}
s:insert{3, 'Ace of Base', 1993}
Important
Please do not close the terminal window
where Tarantool is running – you’ll need it soon.
In order to connect to Tarantool as an administrator, reset the password
for the admin user:
box.schema.user.passwd('pass')
To insert a tuple into a space, use Insert:
resp, err = conn.Insert("tester", []interface{}{4, "ABBA", 1972})
This inserts the tuple (4, "ABBA", 1972) into a space named tester.
The response code and data are available in the
tarantool.Response
structure:
code := resp.Code
data := resp.Data
To select a tuple from a space, use
Select:
resp, err = conn.Select("tester", "primary", 0, 1, tarantool.IterEq, []interface{}{4})
This selects a tuple by the primary key with offset = 0 and limit = 1
from a space named tester (in our example, this is the index named primary,
based on the id field of each tuple).
Next, select tuples by a secondary key.
resp, err = conn.Select("tester", "secondary", 0, 1, tarantool.IterEq, []interface{}{"ABBA"})
Finally, it would be nice to select all the tuples in a space. But there is no
one-liner for this in Go; you would need a script like
this one.
For more examples, see https://github.com/tarantool/go-tarantool#usage
Update a field value using Update:
resp, err = conn.Update("tester", "primary", []interface{}{4}, []interface{}{[]interface{}{"+", 2, 3}})
This increases by 3 the value of field 2 in the tuple with id = 4.
If a tuple with this id doesn’t exist, Tarantool will return an error.
Now use Replace to totally replace the tuple that matches the
primary key. If a tuple with this primary key doesn’t exist, Tarantool will
do nothing.
resp, err = conn.Replace("tester", []interface{}{4, "New band", 2011})
You can also update the data using Upsert that works similarly
to Update, but creates a new tuple if the old one was not found.
resp, err = conn.Upsert("tester", []interface{}{4, "Another band", 2000}, []interface{}{[]interface{}{"+", 2, 5}})
This increases by 5 the value of the third field in the tuple with id = 4, or
inserts the tuple (4, "Another band", 2000) if a tuple with this id
doesn’t exist.
To delete a tuple, use сonnection.Delete:
resp, err = conn.Delete("tester", "primary", []interface{}{4})
To delete all tuples in a space (or to delete an entire space), use Call.
We’ll focus on this function in more detail in the
next section.
To delete all tuples in a space, call space:truncate:
resp, err = conn.Call("box.space.tester:truncate", []interface{}{})
To delete an entire space, call space:drop.
This requires connecting to Tarantool as the admin user:
resp, err = conn.Call("box.space.tester:drop", []interface{}{})
Executing stored procedures
Switch to the terminal window where Tarantool is running.
Note
If you don’t have a terminal window with remote connection to Tarantool,
check out these guides:
Define a simple Lua function:
function sum(a, b)
return a + b
end
Now we have a Lua function defined in Tarantool. To invoke this function from
go, use Call:
resp, err = conn.Call("sum", []interface{}{2, 3})
To send bare Lua code for execution, use Eval:
resp, err = connection.Eval("return 4 + 5", []interface{}{})
There are two more connectors from the open-source community:
See the feature comparison table of all Go connectors available.
Creating your first Tarantool Cartridge application
Here we’ll walk you through developing a simple cluster application.
First,
set up the development environment.
Next, create an application named myapp. Run:
$ cartridge create --name myapp
This will create a Tarantool Cartridge application in the ./myapp directory,
with a handful of
template files and directories
inside.
Go inside and make a dry run:
$ cd ./myapp
$ cartridge build
$ cartridge start
This will build the application locally, start 5 instances of Tarantool
and a stateboard (state provider), and
run the application as it is, with no business logic yet.
Why 5 instances and a stateboard? See the instances.yml file in your application directory.
It contains the configuration of all instances
that you can use in the cluster. By default, it defines configuration for 5
Tarantool instances and a stateboard.
---
myapp.router:
advertise_uri: localhost:3301
http_port: 8081
myapp.s1-master:
advertise_uri: localhost:3302
http_port: 8082
myapp.s1-replica:
advertise_uri: localhost:3303
http_port: 8083
myapp.s2-master:
advertise_uri: localhost:3304
http_port: 8084
myapp.s2-replica:
advertise_uri: localhost:3305
http_port: 8085
myapp-stateboard:
listen: localhost:4401
password: passwd
You can already see these instances in the cluster management web interface at
http://localhost:8081 (here 8081 is the HTTP port of the first instance
specified in instances.yml).
Okay, press Ctrl + C to stop the cluster for a while.
Now it’s time to add some business logic to your application.
This will be an evergreen “Hello world!”” – just to keep things simple.
Rename the template file app/roles/custom.lua to hello-world.lua.
$ mv app/roles/custom.lua app/roles/hello-world.lua
This will be your role. In Tarantool Cartridge, a role is a Lua module that
implements some instance-specific functions and/or logic.
Further on we’ll show how to add code to a role, build it, enable and test.
There is already some code in the role’s init() function.
local function init(opts) -- luacheck: no unused args
-- if opts.is_master then
-- end
local httpd = assert(cartridge.service_get('httpd'), "Failed to get httpd service")
httpd:route({method = 'GET', path = '/hello'}, function()
return {body = 'Hello world!'}
end)
return true
end
This exports an HTTP endpoint /hello. For example, http://localhost:8081/hello
if you address the first instance from the instances.yml file.
If you open it in a browser after enabling the role (we’ll do it here a bit later),
you’ll see “Hello world!” on the page.
Let’s add some more code there.
local function init(opts) -- luacheck: no unused args
-- if opts.is_master then
-- end
local httpd = cartridge.service_get('httpd')
httpd:route({method = 'GET', path = '/hello'}, function()
return {body = 'Hello world!'}
end)
local log = require('log')
log.info('Hello world!')
return true
end
This writes “Hello, world!” to the console when the role gets enabled,
so you’ll have a chance to spot this. No rocket science.
Next, amend role_name in the “return” section of the hello-world.lua file.
You’ll see this section at the bottom of the file.
This text will be displayed as a label for your role in the cluster management
web interface.
return {
role_name = 'Hello world!',
init = init,
stop = stop,
validate_config = validate_config,
apply_config = apply_config,
-- dependencies = {'cartridge.roles.vshard-router'},
}
The final thing to do before you can run the application is to add your role to
the list of available cluster roles in the init.lua file in the project root directory.
local cartridge = require('cartridge')
local ok, err = cartridge.cfg({
roles = {
'cartridge.roles.vshard-storage',
'cartridge.roles.vshard-router',
'cartridge.roles.metrics',
'app.roles.hello-world',
},
})
Now the cluster will be aware of your role.
Why app.roles.hello-world? By default, the role name here should match the
path from the application root (./myapp) to the role file
(app/roles/hello-world.lua).
Great! Your role is ready. Re-build the application and re-start the cluster now:
$ cartridge build
$ cartridge start
Now all instances are up, but idle, waiting for you to enable roles for them.
Instances (replicas) in a Tarantool Cartridge cluster are organized into
replica sets. Roles are enabled per replica set, so all instances in a
replica set have the same roles enabled.
Let’s create a replica set containing just one instance and enable your role:
Open the cluster management web interface at http://localhost:8081.
Next to the router instance, click Configure.
Check the role Hello world! to enable it. Notice that the role name here
matches the label text that you specified in the role_name parameter in
the hello-world.lua file.
(Optionally) Specify the replica set name, for example
“hello-world-replica-set”.
Click Create replica set and see the newly-created replica set
in the web interface.
Your custom role got enabled. Find the “Hello world!” message in console,
like this:
Finally, open the HTTP endpoint of this instance at
http://localhost:8081/hello and see the reply to your GET request.
Everything is up and running! What’s next?
- Follow the administrator’s guide to set up the rest of the
cluster and try some cool cluster management features – for example, enable failover.
- Check out the Cartridge developer’s guide
and implement more sophisticated business logic for your role.
- Pack your application for easy distribution.
Choose what you like: a DEB or RPM package, a TGZ archive, or a Docker image.
Archives and packages can be deployed with ansible-cartridge.
- Read the Cartridge documentation.
Defining and manipulating data
This section contains guides on performing data operations in Tarantool.
CRUD operation examples
This section shows basic usage scenarios and typical errors for each
data operation in Tarantool:
INSERT,
DELETE,
UPDATE,
UPSERT,
REPLACE, and
SELECT.
Before trying out the examples, you need to bootstrap a Tarantool instance as shown below.
The space_object.insert method accepts a well-formatted tuple.
insert also checks all the keys for duplicates.
space_object.update allows you to update a tuple identified by the primary key.
Similarly to delete, the update method accepts a full key and also an operation to execute.
index_object.update updates a tuple identified by the specified unique index.
space_object.upsert updates an existing tuple or inserts a new one:
- If the existing tuple is found by the primary key,
Tarantool applies the update operation to this tuple
and ignores the new tuple.
- If no existing tuple is found,
Tarantool inserts the new tuple and ignores the update operation.
upsert acts as insert when no existing tuple is found by the primary key.
upsert searches for the existing tuple by the primary index,
not by the secondary index. This can lead to a duplication error
if the tuple violates a secondary index uniqueness.
space_object.replace accepts a well-formatted tuple and searches for the existing tuple
by the primary key of the new tuple:
- If the existing tuple is found, Tarantool deletes it and inserts the new tuple.
- If no existing tuple is found, Tarantool inserts the new tuple.
replace can violate unique constraints, like upsert does.
The space_object.select request searches for a tuple or a set of tuples in the given space
by the primary key.
To search by the specified index, use index_object.select.
These methods work with any keys, including unique and non-unique, full and partial.
If a key is partial, select searches by all keys where the prefix matches the specified key part.
Using box.space functions to read _space tuples
This example illustrates how to look at all the spaces, and for each
display: approximately how many tuples it contains, and the first field of
its first tuple. The function uses the Tarantool’s box.space functions len()
and pairs(). The iteration through the spaces is coded as a scan of the
_space system space, which contains metadata. The third field in
_space contains the space name, so the key instruction
space_name = v[3] means space_name is the space_name field in
the tuple of _space that we’ve just fetched with pairs(). The function
returns a table:
function example()
local tuple_count, space_name, line
local ta = {}
for k, v in box.space._space:pairs() do
space_name = v[3]
if box.space[space_name].index[0] ~= nil then
tuple_count = '1 or more'
else
tuple_count = '0'
end
line = space_name .. ' tuple_count =' .. tuple_count
if tuple_count == '1 or more' then
for k1, v1 in box.space[space_name]:pairs() do
line = line .. '. first field in first tuple = ' .. v1[1]
break
end
end
table.insert(ta, line)
end
return ta
end
The output below shows what happens if you invoke this function:
Using box.space functions to organize a _space tuple
This examples shows how to display field names and field types of a system space –
using metadata to find metadata.
To begin: how can one select the _space tuple that describes _space?
A simple way is to look at the constants in box.schema,
which shows that there is an item named SPACE_ID == 288,
so these statements retrieve the correct tuple:
box.space._space:select{ 288 }
-- or --
box.space._space:select{ box.schema.SPACE_ID }
Another way is to look at the tuples in box.space._index,
which shows that there is a secondary index named ‘name’ for a space
number 288, so this statement also retrieve the correct tuple:
box.space._space.index.name:select{ '_space' }
However, the retrieved tuple is not easy to read:
It looks disorganized because field number 7 has been formatted with recommended
names and data types. How can one get those specific sub-fields? Since it’s
visible that field number 7 is an array of maps, this for loop will do the
organizing:
Using indexes
It is mandatory to create an index for a space before trying to insert
tuples into the space, or select tuples from the space.
The simple index-creation
operation is:
box.space.space-name:create_index('index-name')
This creates a unique TREE index on the first field
of all tuples (often called “Field#1”), which is assumed to be numeric.
A recommended design pattern for a data model is to base primary keys on the
first fields of a tuple. This speeds up tuple comparison due to the specifics of
data storage and the way comparisons are arranged in Tarantool.
The simple SELECT request is:
box.space.space-name:select(value)
This looks for a single tuple via the first index. Since the first index
is always unique, the maximum number of returned tuples will be 1.
You can call select() without arguments, and it will return all tuples.
Be careful! Using select() for huge spaces hangs your instance.
An index definition may also include identifiers of tuple fields
and their expected types. See allowed indexed field types in section
Details about indexed field types:
box.space.space-name:create_index(index-name, {type = 'tree', parts = {{field = 1, type = 'unsigned'}}}
Space definitions and index definitions are stored permanently in Tarantool’s
system spaces _space and _index.
Tip
See full information about creating indexes, such as
how to create a multikey index, an index using the path option, or
how to create a functional index in our reference for
space_object:create_index().
Index operations are automatic: if a data manipulation request changes a tuple,
then it also changes the index keys defined for the tuple.
Let’s create a sample space named tester and
put it in a variable my_space:
Format the created space by specifying field names and types:
Create the primary index (named primary):
This is a primary index based on the id field of each tuple.
Insert some tuples into the space:
Create a secondary index:
Create a multi-part index with three parts:
There are the following SELECT variations:
The search can use comparisons other than equality:
The comparison operators are:
LT for “less than”LE for “less than or equal”GT for “greater”GE for “greater than or equal” .EQ for “equal”,REQ for “reversed equal”
Value comparisons make sense if and only if the index type is TREE.
The iterator types for other types of indexes are slightly different and work
differently. See details in section Iterator types.
Note that we don’t use the name of the index, which means we use primary index here.
This type of search may return more than one tuple. The tuples will be sorted
in descending order by key if the comparison operator is LT or LE or REQ.
Otherwise they will be sorted in ascending order.
The search can use a secondary index.
For a primary-key search, it is optional to specify an index name as
was demonstrated above.
For a secondary-key search, it is mandatory.
Partial key search: The search may be for some key parts starting with
the prefix of the key. Note that partial key searches are available
only in TREE indexes.
The search can be for all fields, using a table as the value:
or the search can be for one field, using a table or a scalar:
Tip
You can also add, drop, or alter the definitions at runtime, with some
restrictions. Read more about index operations in reference for
box.index submodule.
Using sequences
A sequence is a generator of ordered integer values.
As with spaces and indexes, you should specify the sequence name and let
Tarantool generate a unique numeric identifier (sequence ID).
As well, you can specify several options when creating a new sequence.
The options determine what value will be generated whenever the sequence is used.
Options for box.schema.sequence.create()
| Option name |
Type and meaning |
Default |
Examples |
|---|
start |
Integer. The value to generate the first time a sequence is used |
1 |
start=0 |
min |
Integer. Values smaller than this cannot be generated |
1 |
min=-1000 |
max |
Integer. Values larger than this cannot be generated |
9223372036854775807 |
max=0 |
cycle |
Boolean. Whether to start again when values cannot be generated |
false |
cycle=true |
cache |
Integer. The number of values to store in a cache |
0 |
cache=0 |
step |
Integer. What to add to the previous generated value, when generating a new value |
1 |
step=-1 |
if_not_exists |
Boolean. If this is true and a sequence with this name exists already,
ignore other options and use the existing values |
false |
if_not_exists=true |
Once a sequence exists, it can be altered, dropped, reset, forced to generate
the next value, or associated with an index.
Associating a sequence with an index
For an initial example, we generate a sequence named ‘S’.
The result shows that the new sequence has all default values,
except for the two that were specified, min and start.
Then we get the next value, with the next() function.
The result is the same as the start value. If we called next()
again, we would get 6 (because the previous value plus the
step value is 6), and so on.
Then we create a new table and specify that its primary key should be
generated from the sequence.
Then we insert a tuple without specifying a value for the primary key.
The result is a new tuple where the first field has a value of 6.
This arrangement, where the system automatically generates the
values for a primary key, is sometimes called “auto-incrementing”
or “identity”.
For syntax and implementation details, see the reference for
box.schema.sequence.
Getting started with net.box
The tutorial shows how to work with some common net.box methods.
For more information about the net.box module,
check the corresponding module reference.
The sandbox configuration for the tutorial assumes that:
- The Tarantool instance is running on
localhost 127.0.0.1:3301.
- There is a space named
tester with a numeric primary key.
- The space contains a tuple with the key value =
800.
- The current user has read, write, and execute privileges.
Use the commands below for a quick sandbox setup:
box.cfg{listen = 3301}
s = box.schema.space.create('tester')
s:create_index('primary', {type = 'hash', parts = {1, 'unsigned'}})
t = s:insert({800, 'TEST'})
box.schema.user.grant('guest', 'read,write,execute', 'universe')
Creating a net.box connection
First, load the net.box module with the require('net.box') method:
The next step is to create a new connection.
In net.box, self-connection is pre-established.
That is, conn = net_box.connect('localhost:3301') command can be replaced with the conn = net_box.self object call:
Then, make a ping:
Select all tuples in the tester space where the key value is 800:
Insert two tuples into the space:
After the insert, there is one tuple where the key value is 600.
To select this tuple, you can use the get() method.
Unlike the select() command, get() returns only one tuple that satisfies the stated condition.
To update the existing tuple, you can use either update() or upsert().
The update() method can be used for assignment, arithmetic (if the field is numeric),
cutting and pasting fragments of a field, and deleting or inserting a field.
In this tutorial, the update() command is used to update the tuple identified by primary key value = 800.
The operation assigns a new value to the second field in the tuple:
As for the upsert function, if there is an existing tuple that matches the key field of tuple, then the command
has the same effect as update().
Otherwise, the effect is equal to the insert() method.
To delete a tuple where the key value is 600, run the delete() method below:
Then, replace the existing tuple with a new one:
Finally, select all tuples from the space:
In the end, close the connection when it is no longer needed:
Quick start with sharding
For installation instructions, check out the vshard installation manual.
For a pre-configured development cluster, check out the example/ directory in
the vshard repository.
This example includes 5 Tarantool instances and 2 replica sets:
router_1 – a router instancestorage_1_a – a storage instance, the master of the first replica setstorage_1_b – a storage instance, the replica of the first replica setstorage_2_a – a storage instance, the master of the second replica setstorage_2_b – a storage instance, the replica of the second replica set
All instances are managed using the tarantoolctl administrative utility which comes with Tarantool.
Change the directory to example/ and use make to run the development cluster:
$ cd example/
$ make
tarantoolctl stop storage_1_a # stop the first storage instance
Stopping instance storage_1_a...
tarantoolctl stop storage_1_b
<...>
rm -rf data/
tarantoolctl start storage_1_a # start the first storage instance
Starting instance storage_1_a...
Starting configuration of replica 8a274925-a26d-47fc-9e1b-af88ce939412
I am master
Taking on replicaset master role...
Run console at unix/:./data/storage_1_a.control
started
mkdir ./data/storage_1_a
<...>
tarantoolctl start router_1 # start the router
Starting instance router_1...
Starting router configuration
Calling box.cfg()...
<...>
Run console at unix/:./data/router_1.control
started
mkdir ./data/router_1
Waiting cluster to start
echo "vshard.router.bootstrap()" | tarantoolctl enter router_1
connected to unix/:./data/router_1.control
unix/:./data/router_1.control> vshard.router.bootstrap()
---
- true
...
unix/:./data/router_1.control>
tarantoolctl enter router_1 # enter the admin console
connected to unix/:./data/router_1.control
unix/:./data/router_1.control>
Some tarantoolctl commands:
tarantoolctl start router_1 – start the router instancetarantoolctl enter router_1 – enter the admin console
The full list of tarantoolctl commands for managing Tarantool instances is
available in the tarantoolctl reference.
Essential make commands you need to know:
make start – start all Tarantool instancesmake stop – stop all Tarantool instancesmake logcat – show logs from all instancesmake enter – enter the admin console on router_1make clean – clean up all persistent datamake test – run the test suite (you can also run test-run.py in the test directory)make – execute make stop, make clean, make start and make enter
For example, to start all instances, use make start:
$ make start
$ ps x|grep tarantool
46564 ?? Ss 0:00.34 tarantool storage_1_a.lua <running>
46566 ?? Ss 0:00.19 tarantool storage_1_b.lua <running>
46568 ?? Ss 0:00.35 tarantool storage_2_a.lua <running>
46570 ?? Ss 0:00.20 tarantool storage_2_b.lua <running>
46572 ?? Ss 0:00.25 tarantool router_1.lua <running>
To perform commands in the admin console, use the router’s
public API:
The configuration of a simple sharded cluster can look like this:
local cfg = {
memtx_memory = 100 * 1024 * 1024,
bucket_count = 10000,
rebalancer_disbalance_threshold = 10,
rebalancer_max_receiving = 100,
sharding = {
['cbf06940-0790-498b-948d-042b62cf3d29'] = {
replicas = {
['8a274925-a26d-47fc-9e1b-af88ce939412'] = {
uri = 'storage:storage@127.0.0.1:3301',
name = 'storage_1_a',
master = true
},
['3de2e3e1-9ebe-4d0d-abb1-26d301b84633'] = {
uri = 'storage:storage@127.0.0.1:3302',
name = 'storage_1_b'
}
},
},
['ac522f65-aa94-4134-9f64-51ee384f1a54'] = {
replicas = {
['1e02ae8a-afc0-4e91-ba34-843a356b8ed7'] = {
uri = 'storage:storage@127.0.0.1:3303',
name = 'storage_2_a',
master = true
},
['001688c3-66f8-4a31-8e19-036c17d489c2'] = {
uri = 'storage:storage@127.0.0.1:3304',
name = 'storage_2_b'
}
},
},
},
}
This cluster includes one router instance and two storage instances.
Each storage instance includes one master and one replica.
The sharding field defines the logical topology of a sharded Tarantool cluster.
All the other fields are passed to box.cfg() as they are, without modifications.
See the Configuration reference section for details.
On routers, call vshard.router.cfg(cfg):
cfg.listen = 3300
-- Start the database with sharding
vshard = require('vshard')
vshard.router.cfg(cfg)
On storages, call vshard.storage.cfg(cfg, instance_uuid):
-- Get instance name
local MY_UUID = "de0ea826-e71d-4a82-bbf3-b04a6413e417"
-- Call a configuration provider
local cfg = require('localcfg')
-- Start the database with sharding
vshard = require('vshard')
vshard.storage.cfg(cfg, MY_UUID)
vshard.storage.cfg() automatically calls box.cfg() and configures the listen
port and replication parameters.
For a sample configuration, see router.lua and storage.lua in the
example/ directory of the vshard repository.
Launching an application
Using Tarantool as an application server, you can write your own applications.
Tarantool’s native language for writing applications is
Lua, so a typical application would be
a file that contains your Lua script. But you can also write applications
in C or C++.
Note
If you’re new to Lua, we recommend going over the interactive Tarantool
tutorial before proceeding with this chapter. To launch the tutorial, say
tutorial() in Tarantool console:
Let’s create and launch our first Lua application for Tarantool.
Here’s a simplest Lua application, the good old “Hello, world!”:
#!/usr/bin/env tarantool
print('Hello, world!')
We save it in a file. Let it be myapp.lua in the current directory.
Now let’s discuss how we can launch our application with Tarantool.
If we run Tarantool in a Docker container,
the following command will start Tarantool without any application:
$ # create a temporary container and run it in interactive mode
$ docker run --rm -t -i tarantool/tarantool:1
To run Tarantool with our application, we can say:
$ # create a temporary container and
$ # launch Tarantool with our application
$ docker run --rm -t -i \
-v `pwd`/myapp.lua:/opt/tarantool/myapp.lua \
-v /data/dir/on/host:/var/lib/tarantool \
tarantool/tarantool:1 tarantool /opt/tarantool/myapp.lua
Here two resources on the host get mounted in the container:
- our application file (myapp.lua) and
- Tarantool data directory (
/data/dir/on/host).
By convention, the directory for Tarantool application code inside a container
is /opt/tarantool, and the directory for data is /var/lib/tarantool.
Launching a binary program
If we run Tarantool from a package or from a source build, we can launch our application:
- in the script mode,
- as a server application, or
- as a daemon service.
The simplest way is to pass the filename to Tarantool at start:
$ tarantool myapp.lua
Hello, world!
$
Tarantool starts, executes our script in the script mode and exits.
Now let’s turn this script into a server application. We use
box.cfg from Tarantool’s built-in
Lua module to:
- launch the database (a database has a persistent on-disk state, which needs
to be restored after we start an application) and
- configure Tarantool as a server that accepts requests over a TCP port.
We also add some simple database logic, using
space.create() and
create_index() to create a space with a primary
index. We use the function box.once() to make sure that our
logic will be executed only once when the database is initialized for the first
time, so we don’t try to create an existing space or index on each invocation
of the script:
#!/usr/bin/env tarantool
-- Configure database
box.cfg {
listen = 3301
}
box.once("bootstrap", function()
box.schema.space.create('tweedledum')
box.space.tweedledum:create_index('primary',
{ type = 'TREE', parts = {1, 'unsigned'}})
end)
Now we launch our application in the same manner as before:
$ tarantool myapp.lua
Hello, world!
2017-08-11 16:07:14.250 [41436] main/101/myapp.lua C> version 2.1.0-429-g4e5231702
2017-08-11 16:07:14.250 [41436] main/101/myapp.lua C> log level 5
2017-08-11 16:07:14.251 [41436] main/101/myapp.lua I> mapping 1073741824 bytes for tuple arena...
2017-08-11 16:07:14.255 [41436] main/101/myapp.lua I> recovery start
2017-08-11 16:07:14.255 [41436] main/101/myapp.lua I> recovering from `./00000000000000000000.snap'
2017-08-11 16:07:14.271 [41436] main/101/myapp.lua I> recover from `./00000000000000000000.xlog'
2017-08-11 16:07:14.271 [41436] main/101/myapp.lua I> done `./00000000000000000000.xlog'
2017-08-11 16:07:14.272 [41436] main/102/hot_standby I> recover from `./00000000000000000000.xlog'
2017-08-11 16:07:14.274 [41436] iproto/102/iproto I> binary: started
2017-08-11 16:07:14.275 [41436] iproto/102/iproto I> binary: bound to [::]:3301
2017-08-11 16:07:14.275 [41436] main/101/myapp.lua I> done `./00000000000000000000.xlog'
2017-08-11 16:07:14.278 [41436] main/101/myapp.lua I> ready to accept requests
This time, Tarantool executes our script and keeps working as a server,
accepting TCP requests on port 3301. We can see Tarantool in the current
session’s process list:
$ ps | grep "tarantool"
PID TTY TIME CMD
41608 ttys001 0:00.47 tarantool myapp.lua <running>
But the Tarantool instance will stop if we close the current terminal window.
To detach Tarantool and our application from the terminal window, we can launch
it in the daemon mode. To do so, we add some parameters to box.cfg{}:
- background =
true that actually tells
Tarantool to work as a daemon service,
- log =
'dir-name' that tells the Tarantool
daemon where to store its log file (other log settings are available in
Tarantool log module), and
- pid_file =
'file-name' that tells the
Tarantool daemon where to store its pid file.
For example:
box.cfg {
listen = 3301,
background = true,
log = '1.log',
pid_file = '1.pid'
}
We launch our application in the same manner as before:
$ tarantool myapp.lua
Hello, world!
$
Tarantool executes our script, gets detached from the current shell session
(you won’t see it with ps | grep "tarantool") and continues working in the
background as a daemon attached to the global session (with SID = 0):
$ ps -ef | grep "tarantool"
PID SID TIME CMD
42178 0 0:00.72 tarantool myapp.lua <running>
Now that we have discussed how to create and launch a Lua application for
Tarantool, let’s dive deeper into programming practices.
Creating an application
Further we walk you through key programming practices that will give you a good
start in writing Lua applications for Tarantool. We will implement a real microservice
based on Tarantool! It is a backend for a simplified version of
Pokémon Go, a location-based
augmented reality game launched in mid-2016.
In this game, players use the GPS capability of a mobile device to locate, catch,
battle, and train virtual monsters called “pokémon” that appear on the screen as
if they were in the same real-world location as the player.
To stay within the walk-through format, let’s narrow the original gameplay as
follows. We have a map with pokémon spawn locations. Next, we have multiple
players who can send catch-a-pokémon requests to the server (which runs our
Tarantool microservice). The server responds whether the
pokémon is caught or not, increases the player’s pokémon counter if yes,
and triggers the respawn-a-pokémon method that spawns a new pokémon at the same
location in a while.
We leave client-side applications outside the scope of this story. However, we
promise a mini-demo in the end to simulate real users and give us some fun.
Follow these topics to implement our application:
Modules, rocks and applications
To make our game logic available to other developers and Lua applications, let’s
put it into a Lua module.
A module (called “rock” in Lua) is an optional library which enhances
Tarantool functionality. So, we can install our logic as a module in Tarantool
and use it from any Tarantool application or module. Like applications, modules
in Tarantool can be written in Lua (rocks), C or C++.
Modules are good for two things:
- easier code management (reuse, packaging, versioning), and
- hot code reload without restarting the Tarantool instance.
Technically, a module is a file with source code that exports its functions in
an API. For example, here is a Lua module named mymodule.lua that exports
one function named myfun:
local exports = {}
exports.myfun = function(input_string)
print('Hello', input_string)
end
return exports
To launch the function myfun() – from another module, from a Lua application,
or from Tarantool itself, – we need to save this module as a file, then load
this module with the require() directive and call the exported function.
For example, here’s a Lua application that uses myfun() function from
mymodule.lua module:
-- loading the module
local mymodule = require('mymodule')
-- calling myfun() from within test() function
local test = function()
mymodule.myfun()
end
A thing to remember here is that the require() directive takes load paths
to Lua modules from the package.path variable. This is a semicolon-separated
string, where a question mark is used to interpolate the module name. By default,
this variable contains system-wide Lua paths and the working directory.
But if we put our modules inside a specific folder (e.g. scripts/), we need
to add this folder to package.path before any calls to require():
package.path = 'scripts/?.lua;' .. package.path
For our microservice, a simple and convenient solution would be to put all
methods in a Lua module (say pokemon.lua) and to write a Lua application
(say game.lua) that initializes the gaming environment and starts the game
loop.
Now let’s get down to implementation details. In our game, we need three entities:
- map, which is an array of pokémons with coordinates of respawn locations;
in this version of the game, let a location be a rectangle identified with two
points, upper-left and lower-right;
- player, which has an ID, a name, and coordinates of the player’s location
point;
- pokémon, which has the same fields as the player, plus a status
(active/inactive, that is present on the map or not) and a catch probability
(well, let’s give our pokémons a chance to escape :-) )
We’ll store these entities as tuples in Tarantool spaces. But to deliver our
backend application as a microservice, the good practice would be to send/receive
our data in the universal JSON format, thus using Tarantool as a document storage.
Avro schemas
To store JSON data as tuples, we will apply a savvy practice which reduces data
footprint and ensures all stored documents are valid. We will use Tarantool
module avro-schema which checks
the schema of a JSON document and converts it to a Tarantool tuple. The tuple
will contain only field values, and thus take a lot less space than the original
document. In avro-schema terms, converting JSON documents to tuples is
“flattening”, and restoring the original documents is “unflattening”.
First you need to
install
the module with tarantoolctl rocks install avro-schema.
Further usage is quite straightforward:
- For each entity, we need to define a schema in
Apache Avro schema syntax,
where we list the entity’s fields with their names and
Avro data types.
- At initialization, we call
avro-schema.create() that creates objects
in memory for all schema entities, and compile() that generates
flatten/unflatten methods for each entity.
- Further on, we just call flatten/unflatten methods for a respective entity
on receiving/sending the entity’s data.
Here’s what our schema definitions for the player and pokémon entities look like:
local schema = {
player = {
type="record",
name="player_schema",
fields={
{name="id", type="long"},
{name="name", type="string"},
{
name="location",
type= {
type="record",
name="player_location",
fields={
{name="x", type="double"},
{name="y", type="double"}
}
}
}
}
},
pokemon = {
type="record",
name="pokemon_schema",
fields={
{name="id", type="long"},
{name="status", type="string"},
{name="name", type="string"},
{name="chance", type="double"},
{
name="location",
type= {
type="record",
name="pokemon_location",
fields={
{name="x", type="double"},
{name="y", type="double"}
}
}
}
}
}
}
And here’s how we create and compile our entities at initialization:
-- load avro-schema module with require()
local avro = require('avro_schema')
-- create models
local ok_m, pokemon = avro.create(schema.pokemon)
local ok_p, player = avro.create(schema.player)
if ok_m and ok_p then
-- compile models
local ok_cm, compiled_pokemon = avro.compile(pokemon)
local ok_cp, compiled_player = avro.compile(player)
if ok_cm and ok_cp then
-- start the game
<...>
else
log.error('Schema compilation failed')
end
else
log.info('Schema creation failed')
end
return false
As for the map entity, it would be an overkill to introduce a schema for it,
because we have only one map in the game, it has very few fields, and – which
is most important – we use the map only inside our logic, never exposing it
to external users.
Next, we need methods to implement the game logic. To simulate object-oriented
programming in our Lua code, let’s store all Lua functions and shared variables
in a single local variable (let’s name it as game). This will allow us to
address functions or variables from within our module as self.func_name or
self.var_name. Like this:
local game = {
-- a local variable
num_players = 0,
-- a method that prints a local variable
hello = function(self)
print('Hello! Your player number is ' .. self.num_players .. '.')
end,
-- a method that calls another method and returns a local variable
sign_in = function(self)
self.num_players = self.num_players + 1
self:hello()
return self.num_players
end
}
In OOP terms, we can now regard local variables inside game as object fields,
and local functions as object methods.
Note
In this manual, Lua examples use local variables. Use global
variables with caution, since the module’s users may be unaware of them.
To enable/disable the use of undeclared global variables in your Lua code,
use Tarantool’s strict module.
So, our game module will have the following methods:
catch() to calculate whether the pokémon was caught (besides the
coordinates of both the player and pokémon, this method will apply
a probability factor, so not every pokémon within the player’s reach
will be caught);respawn() to add missing pokémons to the map, say, every 60 seconds
(we assume that a frightened pokémon runs away, so we remove a pokémon from
the map on any catch attempt and add it back to the map in a while);notify() to log information about caught pokémons (like
“Player 1 caught pokémon A”);start() to initialize the game (it will create database spaces, create
and compile avro schemas, and launch respawn()).
Besides, it would be convenient to have methods for working with Tarantool
storage. For example:
add_pokemon() to add a pokémon to the database, andmap() to populate the map with all pokémons stored in Tarantool.
We’ll need these two methods primarily when initializing our game, but we can
also call them later, for example to test our code.
Bootstrapping a database
Let’s discuss game initialization. In start() method, we need to populate
Tarantool spaces with pokémon data. Why not keep all game data in memory?
Why use a database? The answer is: persistence.
Without a database, we risk losing data on power outage, for example.
But if we store our data in an in-memory database, Tarantool takes care to
persist it on disk whenever it’s changed. This gives us one more benefit:
quick startup in case of failure.
Tarantool has a smart algorithm that quickly
loads all data from disk into memory on startup, so the warm-up takes little time.
We’ll be using functions from Tarantool built-in box module:
box.schema.create_space('pokemons') to create a space named pokemon for
storing information about pokémons (we don’t create a similar space for players,
because we intend to only send/receive player information via API calls, so we
needn’t store it);box.space.pokemons:create_index('primary', {type = 'hash', parts = {1, 'unsigned'}})
to create a primary HASH index by pokémon ID;box.space.pokemons:create_index('status', {type = 'tree', parts = {2, 'str'}})
to create a secondary TREE index by pokémon status.
Notice the parts = argument in the index specification. The pokémon ID is
the first field in a Tarantool tuple since it’s the first member of the respective
Avro type. So does the pokémon status. The actual JSON document may have ID or
status fields at any position of the JSON map.
The implementation of start() method looks like this:
-- create game object
start = function(self)
-- create spaces and indexes
box.once('init', function()
box.schema.create_space('pokemons')
box.space.pokemons:create_index(
"primary", {type = 'hash', parts = {1, 'unsigned'}}
)
box.space.pokemons:create_index(
"status", {type = "tree", parts = {2, 'str'}}
)
end)
-- create models
local ok_m, pokemon = avro.create(schema.pokemon)
local ok_p, player = avro.create(schema.player)
if ok_m and ok_p then
-- compile models
local ok_cm, compiled_pokemon = avro.compile(pokemon)
local ok_cp, compiled_player = avro.compile(player)
if ok_cm and ok_cp then
-- start the game
<...>
else
log.error('Schema compilation failed')
end
else
log.info('Schema creation failed')
end
return false
end
GIS
Now let’s discuss catch(), which is the main method in our gaming logic.
Here we receive the player’s coordinates and the target pokémon’s ID number,
and we need to answer whether the player has actually caught the pokémon or not
(remember that each pokémon has a chance to escape).
First thing, we validate the received player data against its
Avro schema. And we check whether such a pokémon
exists in our database and is displayed on the map (the pokémon must have the
active status):
catch = function(self, pokemon_id, player)
-- check player data
local ok, tuple = self.player_model.flatten(player)
if not ok then
return false
end
-- get pokemon data
local p_tuple = box.space.pokemons:get(pokemon_id)
if p_tuple == nil then
return false
end
local ok, pokemon = self.pokemon_model.unflatten(p_tuple)
if not ok then
return false
end
if pokemon.status ~= self.state.ACTIVE then
return false
end
-- more catch logic to follow
<...>
end
Next, we calculate the answer: caught or not.
To work with geographical coordinates, we use Tarantool
gis module.
To keep things simple, we don’t load any specific map, assuming that we deal with
a world map. And we do not validate incoming coordinates, assuming again that all
received locations are within the planet Earth.
We use two geo-specific variables:
wgs84, which stands for the latest revision of the World Geodetic System
standard, WGS84.
Basically, it comprises a standard coordinate system for the Earth and
represents the Earth as an ellipsoid.nationalmap, which stands for the
US National Atlas Equal Area. This is a projected
coordinates system based on WGS84. It gives us a zero base for location
projection and allows positioning our players and pokémons in meters.
Both these systems are listed in the EPSG Geodetic Parameter Registry, where each
system has a unique number. In our code, we assign these listing numbers to
respective variables:
wgs84 = 4326,
nationalmap = 2163,
For our game logic, we need one more variable, catch_distance, which defines
how close a player must get to a pokémon before trying to catch it. Let’s set
the distance to 100 meters.
Now we’re ready to calculate the answer. We need to project the current location
of both player (p_pos) and pokémon (m_pos) on the map, check whether the
player is close enough to the pokémon (using catch_distance), and calculate
whether the player has caught the pokémon (here we generate some random value and
let the pokémon escape if the random value happens to be less than 100 minus
pokémon’s chance value):
-- project locations
local m_pos = gis.Point(
{pokemon.location.x, pokemon.location.y}, self.wgs84
):transform(self.nationalmap)
local p_pos = gis.Point(
{player.location.x, player.location.y}, self.wgs84
):transform(self.nationalmap)
-- check catch distance condition
if p_pos:distance(m_pos) > self.catch_distance then
return false
end
-- try to catch pokemon
local caught = math.random(100) >= 100 - pokemon.chance
if caught then
-- update and notify on success
box.space.pokemons:update(
pokemon_id, {{'=', self.STATUS, self.state.CAUGHT}}
)
self:notify(player, pokemon)
end
return caught
Index iterators
By our gameplay, all caught pokémons are returned back to the map. We do this
for all pokémons on the map every 60 seconds using respawn() method.
We iterate through pokémons by status using Tarantool index iterator function
index_object:pairs() and reset the statuses of all
“caught” pokémons back to “active” using box.space.pokemons:update().
respawn = function(self)
fiber.name('Respawn fiber')
for _, tuple in box.space.pokemons.index.status:pairs(
self.state.CAUGHT) do
box.space.pokemons:update(
tuple[self.ID],
{{'=', self.STATUS, self.state.ACTIVE}}
)
end
end
For readability, we introduce named fields:
ID = 1,
STATUS = 2,
The complete implementation of start() now looks like this:
-- create game object
start = function(self)
-- create spaces and indexes
box.once('init', function()
box.schema.create_space('pokemons')
box.space.pokemons:create_index(
"primary", {type = 'hash', parts = {1, 'unsigned'}}
)
box.space.pokemons:create_index(
"status", {type = "tree", parts = {2, 'str'}}
)
end)
-- create models
local ok_m, pokemon = avro.create(schema.pokemon)
local ok_p, player = avro.create(schema.player)
if ok_m and ok_p then
-- compile models
local ok_cm, compiled_pokemon = avro.compile(pokemon)
local ok_cp, compiled_player = avro.compile(player)
if ok_cm and ok_cp then
-- start the game
self.pokemon_model = compiled_pokemon
self.player_model = compiled_player
self.respawn()
log.info('Started')
return true
else
log.error('Schema compilation failed')
end
else
log.info('Schema creation failed')
end
return false
end
Fibers, yields and cooperative multitasking
But wait! If we launch it as shown above – self.respawn() – the function
will be executed only once, just like all the other methods. But we need to
execute respawn() every 60 seconds. Creating a fiber
is the Tarantool way of making application logic work in the background at all
times.
A fiber is a set of instructions that are executed with
cooperative multitasking:
the instructions contain yield signals, upon which control is passed to another fiber.
Let’s launch respawn() in a fiber to make it work in the background all the time.
To do so, we’ll need to amend respawn():
respawn = function(self)
-- let's give our fiber a name;
-- this will produce neat output in fiber.info()
fiber.name('Respawn fiber')
while true do
for _, tuple in box.space.pokemons.index.status:pairs(
self.state.CAUGHT) do
box.space.pokemons:update(
tuple[self.ID],
{{'=', self.STATUS, self.state.ACTIVE}}
)
end
fiber.sleep(self.respawn_time)
end
end
and call it as a fiber in start():
start = function(self)
-- create spaces and indexes
<...>
-- create models
<...>
-- compile models
<...>
-- start the game
self.pokemon_model = compiled_pokemon
self.player_model = compiled_player
fiber.create(self.respawn, self)
log.info('Started')
-- errors if schema creation or compilation fails
<...>
end
Logging
One more helpful function that we used in start() was log.infо() from
Tarantool log module. We also need this function in
notify() to add a record to the log file on every successful catch:
-- event notification
notify = function(self, player, pokemon)
log.info("Player '%s' caught '%s'", player.name, pokemon.name)
end
We use default Tarantool log settings, so we’ll see the log
output in console when we launch our application in script mode.
Great! We’ve discussed all programming practices used in our Lua module (see
pokemon.lua).
Now let’s prepare the test environment. As planned, we write a Lua application
(see game.lua) to
initialize Tarantool’s database module, initialize our game, call the game loop
and simulate a couple of player requests.
To launch our microservice, we put both the pokemon.lua module and the game.lua
application in the current directory, install all external modules, and launch
the Tarantool instance running our game.lua application (this example is for
Ubuntu):
$ ls
game.lua pokemon.lua
$ sudo apt-get install tarantool-gis
$ sudo apt-get install tarantool-avro-schema
$ tarantool game.lua
Tarantool starts and initializes the database. Then Tarantool executes the demo
logic from game.lua: adds a pokémon named Pikachu (its chance to be caught
is very high, 99.1), displays the current map (it contains one active pokémon,
Pikachu) and processes catch requests from two players. Player1 is located just
near the lonely Pikachu pokémon and Player2 is located far away from it.
As expected, the catch results in this output are “true” for Player1 and “false”
for Player2. Finally, Tarantool displays the current map which is empty, because
Pikachu is caught and temporarily inactive:
$ tarantool game.lua
2017-01-09 20:19:24.605 [6282] main/101/game.lua C> version 1.7.3-43-gf5fa1e1
2017-01-09 20:19:24.605 [6282] main/101/game.lua C> log level 5
2017-01-09 20:19:24.605 [6282] main/101/game.lua I> mapping 1073741824 bytes for tuple arena...
2017-01-09 20:19:24.609 [6282] main/101/game.lua I> initializing an empty data directory
2017-01-09 20:19:24.634 [6282] snapshot/101/main I> saving snapshot `./00000000000000000000.snap.inprogress'
2017-01-09 20:19:24.635 [6282] snapshot/101/main I> done
2017-01-09 20:19:24.641 [6282] main/101/game.lua I> ready to accept requests
2017-01-09 20:19:24.786 [6282] main/101/game.lua I> Started
---
- {'id': 1, 'status': 'active', 'location': {'y': 2, 'x': 1}, 'name': 'Pikachu', 'chance': 99.1}
...
2017-01-09 20:19:24.789 [6282] main/101/game.lua I> Player 'Player1' caught 'Pikachu'
true
false
--- []
...
2017-01-09 20:19:24.789 [6282] main C> entering the event loop
nginx
In the real life, this microservice would work over HTTP. Let’s add
nginx web server to our environment and make a similar
demo. But how do we make Tarantool methods callable via REST API? We use nginx
with Tarantool nginx upstream
module and create one more Lua script
(app.lua) that
exports three of our game methods – add_pokemon(), map() and catch()
– as REST endpoints of the nginx upstream module:
local game = require('pokemon')
box.cfg{listen=3301}
game:start()
-- add, map and catch functions exposed to REST API
function add(request, pokemon)
return {
result=game:add_pokemon(pokemon)
}
end
function map(request)
return {
map=game:map()
}
end
function catch(request, pid, player)
local id = tonumber(pid)
if id == nil then
return {result=false}
end
return {
result=game:catch(id, player)
}
end
An easy way to configure and launch nginx would be to create a Docker container
based on a Docker image
with nginx and the upstream module already installed (see
http/Dockerfile).
We take a standard
nginx.conf,
where we define an upstream with our Tarantool backend running (this is another
Docker container, see details below):
upstream tnt {
server pserver:3301 max_fails=1 fail_timeout=60s;
keepalive 250000;
}
and add some Tarantool-specific parameters (see descriptions in the upstream
module’s README
file):
server {
server_name tnt_test;
listen 80 default deferred reuseport so_keepalive=on backlog=65535;
location = / {
root /usr/local/nginx/html;
}
location /api {
# answers check infinity timeout
tnt_read_timeout 60m;
if ( $request_method = GET ) {
tnt_method "map";
}
tnt_http_rest_methods get;
tnt_http_methods all;
tnt_multireturn_skip_count 2;
tnt_pure_result on;
tnt_pass_http_request on parse_args;
tnt_pass tnt;
}
}
Likewise, we put Tarantool server and all our game logic in a second Docker
container based on the
official Tarantool 1.9 image (see
src/Dockerfile)
and set the container’s default command to tarantool app.lua.
This is the backend.
Non-blocking IO
To test the REST API, we create a new script
(client.lua),
which is similar to our game.lua application, but makes HTTP POST and GET
requests rather than calling Lua functions:
local http = require('curl').http()
local json = require('json')
local URI = os.getenv('SERVER_URI')
local fiber = require('fiber')
local player1 = {
name="Player1",
id=1,
location = {
x=1.0001,
y=2.0003
}
}
local player2 = {
name="Player2",
id=2,
location = {
x=30.123,
y=40.456
}
}
local pokemon = {
name="Pikachu",
chance=99.1,
id=1,
status="active",
location = {
x=1,
y=2
}
}
function request(method, body, id)
local resp = http:request(
method, URI, body
)
if id ~= nil then
print(string.format('Player %d result: %s',
id, resp.body))
else
print(resp.body)
end
end
local players = {}
function catch(player)
fiber.sleep(math.random(5))
print('Catch pokemon by player ' .. tostring(player.id))
request(
'POST', '{"method": "catch",
"params": [1, '..json.encode(player)..']}',
tostring(player.id)
)
table.insert(players, player.id)
end
print('Create pokemon')
request('POST', '{"method": "add",
"params": ['..json.encode(pokemon)..']}')
request('GET', '')
fiber.create(catch, player1)
fiber.create(catch, player2)
-- wait for players
while #players ~= 2 do
fiber.sleep(0.001)
end
request('GET', '')
os.exit()
When you run this script, you’ll notice that both players have equal chances to
make the first attempt at catching the pokémon. In a classical Lua script,
a networked call blocks the script until it’s finished, so the first catch
attempt can only be done by the player who entered the game first. In Tarantool,
both players play concurrently, since all modules are integrated with Tarantool
cooperative multitasking and use
non-blocking I/O.
Indeed, when Player1 makes its first REST call, the script doesn’t block.
The fiber running catch() function on behalf of Player1 issues a non-blocking
call to the operating system and yields control to the next fiber, which happens
to be the fiber of Player2. Player2’s fiber does the same. When the network
response is received, Player1’s fiber is activated by Tarantool cooperative
scheduler, and resumes its work. All Tarantool modules
use non-blocking I/O and are integrated with Tarantool cooperative scheduler.
For module developers, Tarantool provides an API.
For our HTTP test, we create a third container based on the
official Tarantool 1.9 image (see
client/Dockerfile)
and set the container’s default command to tarantool client.lua.
To run this test locally, download our pokemon
project from GitHub and say:
$ docker-compose build
$ docker-compose up
Docker Compose builds and runs all the three containers: pserver (Tarantool
backend), phttp (nginx) and pclient (demo client). You can see log
messages from all these containers in the console, pclient saying that it made
an HTTP request to create a pokémon, made two catch requests, requested the map
(empty since the pokémon is caught and temporarily inactive) and exited:
pclient_1 | Create pokemon
<...>
pclient_1 | {"result":true}
pclient_1 | {"map":[{"id":1,"status":"active","location":{"y":2,"x":1},"name":"Pikachu","chance":99.100000}]}
pclient_1 | Catch pokemon by player 2
pclient_1 | Catch pokemon by player 1
pclient_1 | Player 1 result: {"result":true}
pclient_1 | Player 2 result: {"result":false}
pclient_1 | {"map":[]}
pokemon_pclient_1 exited with code 0
Congratulations! Here’s the end point of our walk-through. As further reading,
see more about installing and
contributing a module.
See also reference on Tarantool modules and
C API, and don’t miss our
Lua cookbook recipes.
Developing with an IDE
You can use IntelliJ IDEA as an IDE to develop and debug Lua applications for
Tarantool.
Download and install the IDE from the
official web-site.
JetBrains provides specialized editions for particular languages:
IntelliJ IDEA (Java), PHPStorm (PHP), PyCharm (Python), RubyMine (Ruby),
CLion (C/C++), WebStorm (Web) and others.
So, download a version that suits your primary programming language.
Tarantool integration is supported for all editions.
Configure the IDE:
Start IntelliJ IDEA.
Click Configure button and select Plugins.
Click Browse repositories.
Install EmmyLua plugin.
Note
Please don’t be confused with Lua plugin, which is less powerful
than EmmyLua.
Restart IntelliJ IDEA.
Click Configure, select Project Defaults and then
Run Configurations.
Find Lua Application in the sidebar at the left.
In Program, type a path to an installed tarantool binary.
By default, this is tarantool or /usr/bin/tarantool on most
platforms.
If you installed tarantool from sources to a custom directory,
please specify the proper path here.
Now IntelliJ IDEA is ready to use with Tarantool.
Create a new Lua project.
Add a new Lua file, for example init.lua.
Write your code, save the file.
To run you application, click Run -> Run in the main menu and select
your source file in the list.
Or click Run -> Debug to start debugging.
Note
To use Lua debugger, please upgrade Tarantool to version
1.7.5-29-gbb6170e4b or later.
Lua cookbook recipes
Here are contributions of Lua programs for some frequent or tricky situations.
You can execute any of these programs by copying the code into a .lua file,
and then entering chmod +x ./program-name.lua
and ./program-name.lua on the terminal.
The first line is a “hashbang”:
This runs Tarantool Lua application server, which should be on the execution
path.
This section contains the following recipes:
Use freely.
See more recipes on Tarantool GitHub.
The standard example of a simple program.
#!/usr/bin/env tarantool
print('Hello, World!')
Use box.once() to initialize a database
(creating spaces) if this is the first time the server has been run.
Then use console.start() to start interactive mode.
#!/usr/bin/env tarantool
-- Configure database
box.cfg {
listen = 3313
}
box.once("bootstrap", function()
box.schema.space.create('tweedledum')
box.space.tweedledum:create_index('primary',
{ type = 'TREE', parts = {1, 'unsigned'}})
end)
require('console').start()
Use the fio module to open, read, and close a file.
#!/usr/bin/env tarantool
local fio = require('fio')
local errno = require('errno')
local f = fio.open('/tmp/xxxx.txt', {'O_RDONLY' })
if not f then
error("Failed to open file: "..errno.strerror())
end
local data = f:read(4096)
f:close()
print(data)
Use the fio module to open, write, and close a file.
#!/usr/bin/env tarantool
local fio = require('fio')
local errno = require('errno')
local f = fio.open('/tmp/xxxx.txt', {'O_CREAT', 'O_WRONLY', 'O_APPEND'},
tonumber('0666', 8))
if not f then
error("Failed to open file: "..errno.strerror())
end
f:write("Hello\n");
f:close()
Use the LuaJIT ffi library to call a C built-in function: printf().
(For help understanding ffi, see the FFI tutorial.)
#!/usr/bin/env tarantool
local ffi = require('ffi')
ffi.cdef[[
int printf(const char *format, ...);
]]
ffi.C.printf("Hello, %s\n", os.getenv("USER"));
Use the LuaJIT ffi library to call a C function: gettimeofday().
This delivers time with millisecond precision, unlike the time function in
Tarantool’s clock module.
#!/usr/bin/env tarantool
local ffi = require('ffi')
ffi.cdef[[
typedef long time_t;
typedef struct timeval {
time_t tv_sec;
time_t tv_usec;
} timeval;
int gettimeofday(struct timeval *t, void *tzp);
]]
local timeval_buf = ffi.new("timeval")
local now = function()
ffi.C.gettimeofday(timeval_buf, nil)
return tonumber(timeval_buf.tv_sec * 1000 + (timeval_buf.tv_usec / 1000))
end
Use the LuaJIT ffi library to call a C library function.
(For help understanding ffi, see the FFI tutorial.)
#!/usr/bin/env tarantool
local ffi = require("ffi")
ffi.cdef[[
unsigned long compressBound(unsigned long sourceLen);
int compress2(uint8_t *dest, unsigned long *destLen,
const uint8_t *source, unsigned long sourceLen, int level);
int uncompress(uint8_t *dest, unsigned long *destLen,
const uint8_t *source, unsigned long sourceLen);
]]
local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")
-- Lua wrapper for compress2()
local function compress(txt)
local n = zlib.compressBound(#txt)
local buf = ffi.new("uint8_t[?]", n)
local buflen = ffi.new("unsigned long[1]", n)
local res = zlib.compress2(buf, buflen, txt, #txt, 9)
assert(res == 0)
return ffi.string(buf, buflen[0])
end
-- Lua wrapper for uncompress
local function uncompress(comp, n)
local buf = ffi.new("uint8_t[?]", n)
local buflen = ffi.new("unsigned long[1]", n)
local res = zlib.uncompress(buf, buflen, comp, #comp)
assert(res == 0)
return ffi.string(buf, buflen[0])
end
-- Simple test code.
local txt = string.rep("abcd", 1000)
print("Uncompressed size: ", #txt)
local c = compress(txt)
print("Compressed size: ", #c)
local txt2 = uncompress(c, #txt)
assert(txt2 == txt)
Use the LuaJIT ffi library to
insert a tuple which has a VARBINARY field.
Note that it is allowed only inside a memtx transaction: when box_insert()
does not yield.
Lua does not have direct support for VARBINARY, so using C
is one way to put in data which in MessagePack is stored as bin
(MP_BIN). If the tuple is retrieved later, field “b” will have type = ‘cdata’.
#!/usr/bin/env tarantool
-- box.cfg{} should be here
s = box.schema.space.create('withdata')
s:format({{"b", "varbinary"}})
s:create_index('pk', {parts = {1, "varbinary"}})
buffer = require('buffer')
ffi = require('ffi')
function varbinary_insert(space, bytes)
local tmpbuf = buffer.ibuf()
local p = tmpbuf:alloc(3 + #bytes)
p[0] = 0x91 -- MsgPack code for "array-1"
p[1] = 0xC4 -- MsgPack code for "bin-8" so up to 256 bytes
p[2] = #bytes
for i, c in pairs(bytes) do p[i + 3 - 1] = c end
ffi.cdef[[int box_insert(uint32_t space_id,
const char *tuple,
const char *tuple_end,
box_tuple_t **result);]]
ffi.C.box_insert(space.id, tmpbuf.rpos, tmpbuf.wpos, nil)
tmpbuf:recycle()
end
varbinary_insert(s, {0xDE, 0xAD, 0xBE, 0xAF})
varbinary_insert(s, {0xFE, 0xED, 0xFA, 0xCE})
-- if successful, Tarantool enters the event loop now
Create Lua tables, and print them.
Notice that for the ‘array’ table the iterator function
is ipairs(), while for the ‘map’ table the iterator function
is pairs(). (ipairs() is faster than pairs(), but pairs()
is recommended for map-like tables or mixed tables.)
The display will look like:
“1 Apple | 2 Orange | 3 Grapefruit | 4 Banana | k3 v3 | k1 v1 | k2 v2”.
#!/usr/bin/env tarantool
array = { 'Apple', 'Orange', 'Grapefruit', 'Banana'}
for k, v in ipairs(array) do print(k, v) end
map = { k1 = 'v1', k2 = 'v2', k3 = 'v3' }
for k, v in pairs(map) do print(k, v) end
Use the ‘#’ operator to get the number of items in an array-like Lua table.
This operation has O(log(N)) complexity.
#!/usr/bin/env tarantool
array = { 1, 2, 3}
print(#array)
count_array_with_nils.lua
Missing elements in arrays, which Lua treats as “nil”s,
cause the simple “#” operator to deliver improper results.
The “print(#t)” instruction will print “4”;
the “print(counter)” instruction will print “3”;
the “print(max)” instruction will print “10”.
Other table functions, such as table.sort(), will
also misbehave when “nils” are present.
#!/usr/bin/env tarantool
local t = {}
t[1] = 1
t[4] = 4
t[10] = 10
print(#t)
local counter = 0
for k,v in pairs(t) do counter = counter + 1 end
print(counter)
local max = 0
for k,v in pairs(t) do if k > max then max = k end end
print(max)
count_array_with_nulls.lua
Use explicit NULL values to avoid the problems caused by Lua’s
nil == missing value behavior. Although json.NULL == nil is
true, all the print instructions in this program will print
the correct value: 10.
#!/usr/bin/env tarantool
local json = require('json')
local t = {}
t[1] = 1; t[2] = json.NULL; t[3]= json.NULL;
t[4] = 4; t[5] = json.NULL; t[6]= json.NULL;
t[6] = 4; t[7] = json.NULL; t[8]= json.NULL;
t[9] = json.NULL
t[10] = 10
print(#t)
local counter = 0
for k,v in pairs(t) do counter = counter + 1 end
print(counter)
local max = 0
for k,v in pairs(t) do if k > max then max = k end end
print(max)
Get the number of elements in a map-like table.
#!/usr/bin/env tarantool
local map = { a = 10, b = 15, c = 20 }
local size = 0
for _ in pairs(map) do size = size + 1; end
print(size)
Use a Lua peculiarity to swap two variables without needing a third variable.
#!/usr/bin/env tarantool
local x = 1
local y = 2
x, y = y, x
print(x, y)
Create a class, create a metatable for the class, create an instance of the class.
Another illustration is at http://lua-users.org/wiki/LuaClassesWithMetatable.
#!/usr/bin/env tarantool
-- define class objects
local myclass_somemethod = function(self)
print('test 1', self.data)
end
local myclass_someothermethod = function(self)
print('test 2', self.data)
end
local myclass_tostring = function(self)
return 'MyClass <'..self.data..'>'
end
local myclass_mt = {
__tostring = myclass_tostring;
__index = {
somemethod = myclass_somemethod;
someothermethod = myclass_someothermethod;
}
}
-- create a new object of myclass
local object = setmetatable({ data = 'data'}, myclass_mt)
print(object:somemethod())
print(object.data)
fiber_producer_and_consumer.lua
Start one fiber for producer and one fiber for consumer.
Use fiber.channel() to exchange data and synchronize.
One can tweak the channel size (ch_size in the program code)
to control the number of simultaneous tasks waiting for processing.
#!/usr/bin/env tarantool
local fiber = require('fiber')
local function consumer_loop(ch, i)
-- initialize consumer synchronously or raise an error()
fiber.sleep(0) -- allow fiber.create() to continue
while true do
local data = ch:get()
if data == nil then
break
end
print('consumed', i, data)
fiber.sleep(math.random()) -- simulate some work
end
end
local function producer_loop(ch, i)
-- initialize consumer synchronously or raise an error()
fiber.sleep(0) -- allow fiber.create() to continue
while true do
local data = math.random()
ch:put(data)
print('produced', i, data)
end
end
local function start()
local consumer_n = 5
local producer_n = 3
-- Create a channel
local ch_size = math.max(consumer_n, producer_n)
local ch = fiber.channel(ch_size)
-- Start consumers
for i=1, consumer_n,1 do
fiber.create(consumer_loop, ch, i)
end
-- Start producers
for i=1, producer_n,1 do
fiber.create(producer_loop, ch, i)
end
end
start()
print('started')
Use socket.tcp_connect()
to connect to a remote host via TCP.
Display the connection details and the result of a GET request.
#!/usr/bin/env tarantool
local s = require('socket').tcp_connect('google.com', 80)
print(s:peer().host)
print(s:peer().family)
print(s:peer().type)
print(s:peer().protocol)
print(s:peer().port)
print(s:write("GET / HTTP/1.0\r\n\r\n"))
print(s:read('\r\n'))
print(s:read('\r\n'))
Use socket.tcp_connect()
to set up a simple TCP server, by creating
a function that handles requests and echos them,
and passing the function to
socket.tcp_server().
This program has been used to test with 100,000 clients,
with each client getting a separate fiber.
#!/usr/bin/env tarantool
local function handler(s, peer)
s:write("Welcome to test server, " .. peer.host .."\n")
while true do
local line = s:read('\n')
if line == nil then
break -- error or eof
end
if not s:write("pong: "..line) then
break -- error or eof
end
end
end
local server, addr = require('socket').tcp_server('localhost', 3311, handler)
Use socket.getaddrinfo() to perform
non-blocking DNS resolution, getting both the AF_INET6 and AF_INET
information for ‘google.com’.
This technique is not always necessary for tcp connections because
socket.tcp_connect()
performs socket.getaddrinfo under the hood,
before trying to connect to the first available address.
#!/usr/bin/env tarantool
local s = require('socket').getaddrinfo('google.com', 'http', { type = 'SOCK_STREAM' })
print('host=',s[1].host)
print('family=',s[1].family)
print('type=',s[1].type)
print('protocol=',s[1].protocol)
print('port=',s[1].port)
print('host=',s[2].host)
print('family=',s[2].family)
print('type=',s[2].type)
print('protocol=',s[2].protocol)
print('port=',s[2].port)
Tarantool does not currently have a udp_server function,
therefore socket_udp_echo.lua is more complicated than
socket_tcp_echo.lua.
It can be implemented with sockets and fibers.
#!/usr/bin/env tarantool
local socket = require('socket')
local errno = require('errno')
local fiber = require('fiber')
local function udp_server_loop(s, handler)
fiber.name("udp_server")
while true do
-- try to read a datagram first
local msg, peer = s:recvfrom()
if msg == "" then
-- socket was closed via s:close()
break
elseif msg ~= nil then
-- got a new datagram
handler(s, peer, msg)
else
if s:errno() == errno.EAGAIN or s:errno() == errno.EINTR then
-- socket is not ready
s:readable() -- yield, epoll will wake us when new data arrives
else
-- socket error
local msg = s:error()
s:close() -- save resources and don't wait GC
error("Socket error: " .. msg)
end
end
end
end
local function udp_server(host, port, handler)
local s = socket('AF_INET', 'SOCK_DGRAM', 0)
if not s then
return nil -- check errno:strerror()
end
if not s:bind(host, port) then
local e = s:errno() -- save errno
s:close()
errno(e) -- restore errno
return nil -- check errno:strerror()
end
fiber.create(udp_server_loop, s, handler) -- start a new background fiber
return s
end
A function for a client that connects to this server could
look something like this …
local function handler(s, peer, msg)
-- You don't have to wait until socket is ready to send UDP
-- s:writable()
s:sendto(peer.host, peer.port, "Pong: " .. msg)
end
local server = udp_server('127.0.0.1', 3548, handler)
if not server then
error('Failed to bind: ' .. errno.strerror())
end
print('Started')
require('console').start()
Use the http module
to get data via HTTP.
#!/usr/bin/env tarantool
local http_client = require('http.client')
local json = require('json')
local r = http_client.get('https://api.frankfurter.app/latest?to=USD%2CRUB')
if r.status ~= 200 then
print('Failed to get currency ', r.reason)
return
end
local data = json.decode(r.body)
print(data.base, 'rate of', data.date, 'is', data.rates.RUB, 'RUB or', data.rates.USD, 'USD')
Use the http module
to send data via HTTP.
#!/usr/bin/env tarantool
local http_client = require('http.client')
local json = require('json')
local data = json.encode({ Key = 'Value'})
local headers = { Token = 'xxxx', ['X-Secret-Value'] = '42' }
local r = http_client.post('http://localhost:8081', data, { headers = headers})
if r.status == 200 then
print 'Success'
end
Use the http rock (which must first be installed)
to turn Tarantool into a web server.
#!/usr/bin/env tarantool
local function handler(self)
return self:render{ json = { ['Your-IP-Is'] = self.peer.host } }
end
local server = require('http.server').new(nil, 8080, {charset = "utf8"}) -- listen *:8080
server:route({ path = '/' }, handler)
server:start()
-- connect to localhost:8080 and see json
Use the http rock (which must first be installed)
to generate HTML pages from templates.
The http
rock has a fairly simple template engine which allows execution
of regular Lua code inside text blocks (like PHP). Therefore there is no need
to learn new languages in order to write templates.
#!/usr/bin/env tarantool
local function handler(self)
local fruits = {'Apple', 'Orange', 'Grapefruit', 'Banana'}
return self:render{ fruits = fruits }
end
local server = require('http.server').new(nil, 8080, {charset = "utf8"}) -- nil means '*'
server:route({ path = '/', file = 'index.html.lua' }, handler)
server:start()
An “HTML” file for this server, including Lua, could look like this
(it would produce “1 Apple | 2 Orange | 3 Grapefruit | 4 Banana”).
Create a templates directory and put this file in it:
<html>
<body>
<table border="1">
% for i,v in pairs(fruits) do
<tr>
<td><%= i %></td>
<td><%= v %></td>
</tr>
% end
</table>
</body>
</html>
In Go, there is no one-liner to select all tuples from a Tarantool space.
Yet you can use a script like this one. Call it on the instance you want to
connect to.
package main
import (
"fmt"
"log"
"github.com/tarantool/go-tarantool"
)
/*
box.cfg{listen = 3301}
box.schema.user.passwd('pass')
s = box.schema.space.create('tester')
s:format({
{name = 'id', type = 'unsigned'},
{name = 'band_name', type = 'string'},
{name = 'year', type = 'unsigned'}
})
s:create_index('primary', { type = 'hash', parts = {'id'} })
s:create_index('scanner', { type = 'tree', parts = {'id', 'band_name'} })
s:insert{1, 'Roxette', 1986}
s:insert{2, 'Scorpions', 2015}
s:insert{3, 'Ace of Base', 1993}
*/
func main() {
conn, err := tarantool.Connect("127.0.0.1:3301", tarantool.Opts{
User: "admin",
Pass: "pass",
})
if err != nil {
log.Fatalf("Connection refused")
}
defer conn.Close()
spaceName := "tester"
indexName := "scanner"
idFn := conn.Schema.Spaces[spaceName].Fields["id"].Id
bandNameFn := conn.Schema.Spaces[spaceName].Fields["band_name"].Id
var tuplesPerRequest uint32 = 2
cursor := []interface{}{}
for {
resp, err := conn.Select(spaceName, indexName, 0, tuplesPerRequest, tarantool.IterGt, cursor)
if err != nil {
log.Fatalf("Failed to select: %s", err)
}
if resp.Code != tarantool.OkCode {
log.Fatalf("Select failed: %s", resp.Error)
}
if len(resp.Data) == 0 {
break
}
fmt.Println("Iteration")
tuples := resp.Tuples()
for _, tuple := range tuples {
fmt.Printf("\t%v\n", tuple)
}
lastTuple := tuples[len(tuples)-1]
cursor = []interface{}{lastTuple[idFn], lastTuple[bandNameFn]}
}
}
Lua tutorials
If you’re new to Lua, we recommend going over the interactive Tarantool
tutorial. To launch the tutorial, run the tutorial() command in the Tarantool console:
Insert one million tuples with a Lua stored procedure
This is an exercise assignment: “Insert one million tuples. Each tuple should
have a constantly-increasing numeric primary-key field and a random alphabetic
10-character string field.”
The purpose of the exercise is to show what Lua functions look like inside
Tarantool. It will be necessary to employ the Lua math library, the Lua string
library, the Tarantool box library, the Tarantool box.tuple library, loops, and
concatenations. It should be easy to follow even for a person who has not used
either Lua or Tarantool before. The only requirement is a knowledge of how other
programming languages work and a memory of the first two chapters of this manual.
But for better understanding, follow the comments and the links, which point to
the Lua manual or to elsewhere in this Tarantool manual. To further enhance
learning, type the statements in with the tarantool client while reading along.
In earlier versions of Tarantool, multi-line functions had to be
enclosed within “delimiters”. They are no longer necessary, and
so they will not be used in this tutorial. However, they are still
supported. Users who wish to use delimiters, or users of
older versions of Tarantool, should check the syntax description for
declaring a delimiter before proceeding.
Create a function that returns a string
We will start by making a function that returns a fixed string, “Hello world”.
function string_function()
return "hello world"
end
The word “function” is a Lua keyword – we’re about to go into Lua. The
function name is string_function. The function has one executable statement,
return "hello world". The string “hello world” is enclosed in double quotes
here, although Lua doesn’t care – one could use single quotes instead. The
word “end” means “this is the end of the Lua function declaration.”
To confirm that the function works, we can say
Sending function-name() means “invoke the Lua function.” The effect is
that the string which the function returns will end up on the screen.
For more about Lua strings see Lua manual chapter 2.4 “Strings” . For more
about functions see Lua manual chapter 5 “Functions”.
The screen now looks like this:
Create a function that calls another function and sets a variable
Now that string_function exists, we can invoke it from another
function.
function main_function()
local string_value
string_value = string_function()
return string_value
end
We begin by declaring a variable “string_value”. The word “local”
means that string_value appears only in main_function. If we didn’t use
“local” then string_value would be visible everywhere - even by other
users using other clients connected to this server instance! Sometimes that’s a very
desirable feature for inter-client communication, but not this time.
Then we assign a value to string_value, namely, the result of
string_function(). Soon we will invoke main_function() to check that it
got the value.
For more about Lua variables see Lua manual chapter 4.2 “Local Variables and Blocks” .
The screen now looks like this:
Modify the function so it returns a one-letter random string
Now that it’s a bit clearer how to make a variable, we can change
string_function() so that, instead of returning a fixed literal
“Hello world”, it returns a random letter between ‘A’ and ‘Z’.
function string_function()
local random_number
local random_string
random_number = math.random(65, 90)
random_string = string.char(random_number)
return random_string
end
It is not necessary to destroy the old string_function() contents, they’re
simply overwritten. The first assignment invokes a random-number function
in Lua’s math library; the parameters mean “the number must be an integer
between 65 and 90.” The second assignment invokes an integer-to-character
function in Lua’s string library; the parameter is the code point of the
character. Luckily the ASCII value of ‘A’ is 65 and the ASCII value of ‘Z’
is 90 so the result will always be a letter between A and Z.
For more about Lua math-library functions see Lua users
“Math Library Tutorial”.
For more about Lua string-library functions see Lua users
“String Library Tutorial” .
Once again the string_function() can be invoked from main_function() which
can be invoked with main_function().
The screen now looks like this:
… Well, actually it won’t always look like this because math.random()
produces random numbers. But for the illustration purposes it won’t matter
what the random string values are.
Modify the function so it returns a ten-letter random string
Now that it’s clear how to produce one-letter random strings, we can reach our
goal of producing a ten-letter string by concatenating ten one-letter strings,
in a loop.
function string_function()
local random_number
local random_string
random_string = ""
for x = 1,10,1 do
random_number = math.random(65, 90)
random_string = random_string .. string.char(random_number)
end
return random_string
end
The words “for x = 1,10,1” mean “start with x equals 1, loop until x equals 10,
increment x by 1 for each iteration.” The symbol “..” means “concatenate”, that
is, add the string on the right of the “..” sign to the string on the left of
the “..” sign. Since we start by saying that random_string is “” (a blank
string), the end result is that random_string has 10 random letters. Once
again the string_function() can be invoked from main_function() which
can be invoked with main_function().
For more about Lua loops see Lua manual chapter 4.3.4 “Numeric for”.
The screen now looks like this:
Make a tuple out of a number and a string
Now that it’s clear how to make a 10-letter random string, it’s possible to
make a tuple that contains a number and a 10-letter random string, by invoking
a function in Tarantool’s library of Lua functions.
function main_function()
local string_value, t
string_value = string_function()
t = box.tuple.new({1, string_value})
return t
end
Once this is done, t will be the value of a new tuple which has two fields.
The first field is numeric: 1. The second field is a random string. Once again
the string_function() can be invoked from main_function() which can be
invoked with main_function().
For more about Tarantool tuples see Tarantool manual section Submodule box.tuple.
The screen now looks like this:
Modify main_function to insert a tuple into the database
Now that it’s clear how to make a tuple that contains a number and a 10-letter
random string, the only trick remaining is putting that tuple into tester.
Remember that tester is the first space that was defined in the sandbox, so
it’s like a database table.
function main_function()
local string_value, t
string_value = string_function()
t = box.tuple.new({1,string_value})
box.space.tester:replace(t)
end
The new line here is box.space.tester:replace(t). The name contains
‘tester’ because the insertion is going to be to tester. The second parameter
is the tuple value. To be perfectly correct we could have said
box.space.tester:insert(t) here, rather than box.space.tester:replace(t),
but “replace” means “insert even if there is already a tuple whose primary-key
value is a duplicate”, and that makes it easier to re-run the exercise even if
the sandbox database isn’t empty. Once this is done, tester will contain a tuple
with two fields. The first field will be 1. The second field will be a random
10-letter string. Once again the string_function() can be invoked from
main_function() which can be invoked with main_function(). But
main_function() won’t tell the whole story, because it does not return t, it
only puts t into the database. To confirm that something got inserted, we’ll use
a SELECT request.
main_function()
box.space.tester:select{1}
For more about Tarantool insert and replace calls, see Tarantool manual section
Submodule box.space,
space_object:insert(), and
space_object:replace().
The screen now looks like this:
Modify main_function to insert a million tuples into the database
Now that it’s clear how to insert one tuple into the database, it’s no big deal
to figure out how to scale up: instead of inserting with a literal value = 1
for the primary key, insert with a variable value = between 1 and 1 million, in
a loop. Since we already saw how to loop, that’s a simple thing. The only extra
wrinkle that we add here is a timing function.
function main_function()
local string_value, t
for i = 1,1000000,1 do
string_value = string_function()
t = box.tuple.new({i,string_value})
box.space.tester:replace(t)
end
end
start_time = os.clock()
main_function()
end_time = os.clock()
'insert done in ' .. end_time - start_time .. ' seconds'
The standard Lua function
os.clock()
will return the number of CPU seconds since the
start. Therefore, by getting start_time = number of seconds just before the
inserting, and then getting end_time = number of seconds just after the
inserting, we can calculate (end_time - start_time) = elapsed time in seconds.
We will display that value by putting it in a request without any assignments,
which causes Tarantool to send the value to the client, which prints it. (Lua’s
answer to the C printf() function, which is print(), will also work.)
For more on Lua os.clock() see Lua manual chapter 22.1 “Date and Time”.
For more on Lua print() see Lua manual chapter 5 “Functions”.
Since this is the grand finale, we will redo the final versions of all the
necessary requests: the request that
created string_function(), the request that created main_function(),
and the request that invokes main_function().
function string_function()
local random_number
local random_string
random_string = ""
for x = 1,10,1 do
random_number = math.random(65, 90)
random_string = random_string .. string.char(random_number)
end
return random_string
end
function main_function()
local string_value, t
for i = 1,1000000,1 do
string_value = string_function()
t = box.tuple.new({i,string_value})
box.space.tester:replace(t)
end
end
start_time = os.clock()
main_function()
end_time = os.clock()
'insert done in ' .. end_time - start_time .. ' seconds'
The screen now looks like this:
What has been shown is that Lua functions are quite expressive (in fact one can
do more with Tarantool’s Lua stored procedures than one can do with stored
procedures in some SQL DBMSs), and that it’s straightforward to combine
Lua-library functions and Tarantool-library functions.
What has also been shown is that inserting a million tuples took 37 seconds. The
host computer was a Linux laptop. By changing wal_mode to ‘none’ before
running the test, one can reduce the elapsed time to 4 seconds.
Sum a JSON field for all tuples
This is an exercise assignment: “Assume that inside every tuple there is a
string formatted as JSON. Inside that string there is a JSON numeric field.
For each tuple, find the numeric field’s value and add it to a ‘sum’ variable.
At end, return the ‘sum’ variable.” The purpose of the exercise is to get
experience in one way to read and process tuples.
1json = require('json')
2function sum_json_field(field_name)
3 local v, t, sum, field_value, is_valid_json, lua_table
4 sum = 0
5 for v, t in box.space.tester:pairs() do
6 is_valid_json, lua_table = pcall(json.decode, t[2])
7 if is_valid_json then
8 field_value = lua_table[field_name]
9 if type(field_value) == "number" then sum = sum + field_value end
10 end
11 end
12 return sum
13end
LINE 3: WHY “LOCAL”. This line declares all the variables that will be used in
the function. Actually it’s not necessary to declare all variables at the start,
and in a long function it would be better to declare variables just before using
them. In fact it’s not even necessary to declare variables at all, but an
undeclared variable is “global”. That’s not desirable for any of the variables
that are declared in line 1, because all of them are for use only within the function.
LINE 5: WHY “PAIRS()”. Our job is to go through all the rows and there are two
ways to do it: with box.space.space_object:pairs() or with
variable = select(...) followed by for i, n, 1 do some-function(variable[i]) end.
We preferred pairs() for this example.
LINE 5: START THE MAIN LOOP. Everything inside this “for” loop will be
repeated as long as there is another index key. A tuple is fetched and can be
referenced with variable t.
LINE 6: WHY “PCALL”. If we simply said lua_table = json.decode(t[2])), then
the function would abort with an error if it encountered something wrong with the
JSON string - a missing colon, for example. By putting the function inside “pcall”
(protected call), we’re saying: we want to intercept that sort of error, so if
there’s a problem just set is_valid_json = false and we will know what to do
about it later.
LINE 6: MEANING. The function is json.decode which means decode a JSON
string, and the parameter is t[2] which is a reference to a JSON string. There’s
a bit of hard coding here, we’re assuming that the second field in the tuple is
where the JSON string was inserted. For example, we’re assuming a tuple looks like
field[1]: 444
field[2]: '{"Hello": "world", "Quantity": 15}'
meaning that the tuple’s first field, the primary key field, is a number while
the tuple’s second field, the JSON string, is a string. Thus the entire statement
means “decode t[2] (the tuple’s second field) as a JSON string; if there’s an
error set is_valid_json = false; if there’s no error set is_valid_json = true and
set lua_table = a Lua table which has the decoded string”.
LINE 8. At last we are ready to get the JSON field value from the Lua table that
came from the JSON string. The value in field_name, which is the parameter for the
whole function, must be a name of a JSON field. For example, inside the JSON string
'{"Hello": "world", "Quantity": 15}', there are two JSON fields: “Hello” and
“Quantity”. If the whole function is invoked with sum_json_field("Quantity"),
then field_value = lua_table[field_name] is effectively the same as
field_value = lua_table["Quantity"] or even field_value = lua_table.Quantity.
Those are just three different ways of saying: for the Quantity field in the Lua table,
get the value and put it in variable field_value.
LINE 9: WHY “IF”. Suppose that the JSON string is well formed but the JSON field
is not a number, or is missing. In that case, the function would be aborted when
there was an attempt to add it to the sum. By first checking
type(field_value) == "number", we avoid that abortion. Anyone who knows that
the database is in perfect shape can skip this kind of thing.
And the function is complete. Time to test it. Starting with an empty database,
defined the same way as the sandbox database in our
“Getting started” exercises,
-- if tester is left over from some previous test, destroy it
box.space.tester:drop()
box.schema.space.create('tester')
box.space.tester:create_index('primary', {parts = {1, 'unsigned'}})
then add some tuples where the first field is a number and the second
field is a string.
box.space.tester:insert{444, '{"Item": "widget", "Quantity": 15}'}
box.space.tester:insert{445, '{"Item": "widget", "Quantity": 7}'}
box.space.tester:insert{446, '{"Item": "golf club", "Quantity": "sunshine"}'}
box.space.tester:insert{447, '{"Item": "waffle iron", "Quantit": 3}'}
Since this is a test, there are deliberate errors. The “golf club” and the
“waffle iron” do not have numeric Quantity fields, so must be ignored.
Therefore the real sum of the Quantity field in the JSON strings should be:
15 + 7 = 22.
Invoke the function with sum_json_field("Quantity").
It works. We’ll just leave, as exercises for future improvement, the possibility
that the “hard coding” assumptions could be removed, that there might have to be
an overflow check if some field values are huge, and that the function should
contain a yield instruction if the count of tuples is huge.
Here is a generic function which takes a field identifier
and a search pattern, and returns all tuples that match.
* The field must be the first field of a TREE index.
* The function will use Lua pattern matching,
which allows “magic characters” in regular expressions.
* The initial characters in the pattern, as far as the
first magic character, will be used as an index search key.
For each tuple that is found via the index, there will be
a match of the whole pattern.
* To be cooperative,
the function should yield after every
10 tuples, unless there is a reason to delay yielding.
With this function, we can take advantage of Tarantool’s indexes
for speed, and take advantage of Lua’s pattern matching for flexibility.
It does everything that an SQL
LIKE search can do, and far more.
Read the following Lua code to see how it works.
The comments that begin with “SEE NOTE …” refer to long
explanations that follow the code.
function indexed_pattern_search(space_name, field_no, pattern)
-- SEE NOTE #1 "FIND AN APPROPRIATE INDEX"
if (box.space[space_name] == nil) then
print("Error: Failed to find the specified space")
return nil
end
local index_no = -1
for i=0,box.schema.INDEX_MAX,1 do
if (box.space[space_name].index[i] == nil) then break end
if (box.space[space_name].index[i].type == "TREE"
and box.space[space_name].index[i].parts[1].fieldno == field_no
and (box.space[space_name].index[i].parts[1].type == "scalar"
or box.space[space_name].index[i].parts[1].type == "string")) then
index_no = i
break
end
end
if (index_no == -1) then
print("Error: Failed to find an appropriate index")
return nil
end
-- SEE NOTE #2 "DERIVE INDEX SEARCH KEY FROM PATTERN"
local index_search_key = ""
local index_search_key_length = 0
local last_character = ""
local c = ""
local c2 = ""
for i=1,string.len(pattern),1 do
c = string.sub(pattern, i, i)
if (last_character ~= "%") then
if (c == '^' or c == "$" or c == "(" or c == ")" or c == "."
or c == "[" or c == "]" or c == "*" or c == "+"
or c == "-" or c == "?") then
break
end
if (c == "%") then
c2 = string.sub(pattern, i + 1, i + 1)
if (string.match(c2, "%p") == nil) then break end
index_search_key = index_search_key .. c2
else
index_search_key = index_search_key .. c
end
end
last_character = c
end
index_search_key_length = string.len(index_search_key)
if (index_search_key_length < 3) then
print("Error: index search key " .. index_search_key .. " is too short")
return nil
end
-- SEE NOTE #3 "OUTER LOOP: INITIATE"
local result_set = {}
local number_of_tuples_in_result_set = 0
local previous_tuple_field = ""
while true do
local number_of_tuples_since_last_yield = 0
local is_time_for_a_yield = false
-- SEE NOTE #4 "INNER LOOP: ITERATOR"
for _,tuple in box.space[space_name].index[index_no]:
pairs(index_search_key,{iterator = box.index.GE}) do
-- SEE NOTE #5 "INNER LOOP: BREAK IF INDEX KEY IS TOO GREAT"
if (string.sub(tuple[field_no], 1, index_search_key_length)
> index_search_key) then
break
end
-- SEE NOTE #6 "INNER LOOP: BREAK AFTER EVERY 10 TUPLES -- MAYBE"
number_of_tuples_since_last_yield = number_of_tuples_since_last_yield + 1
if (number_of_tuples_since_last_yield >= 10
and tuple[field_no] ~= previous_tuple_field) then
index_search_key = tuple[field_no]
is_time_for_a_yield = true
break
end
previous_tuple_field = tuple[field_no]
-- SEE NOTE #7 "INNER LOOP: ADD TO RESULT SET IF PATTERN MATCHES"
if (string.match(tuple[field_no], pattern) ~= nil) then
number_of_tuples_in_result_set = number_of_tuples_in_result_set + 1
result_set[number_of_tuples_in_result_set] = tuple
end
end
-- SEE NOTE #8 "OUTER LOOP: BREAK, OR YIELD AND CONTINUE"
if (is_time_for_a_yield ~= true) then
break
end
require('fiber').yield()
end
return result_set
end
NOTE #1 “FIND AN APPROPRIATE INDEX”
The caller has passed space_name (a string) and field_no (a number).
The requirements are:
(a) index type must be “TREE” because for other index types
(HASH, BITSET, RTREE) a search with iterator=GE
will not return strings in order by string value;
(b) field_no must be the first index part;
(c) the field must contain strings, because for other data types
(such as “unsigned”) pattern searches are not possible;
If these requirements are not met by any index, then
print an error message and return nil.
NOTE #2 “DERIVE INDEX SEARCH KEY FROM PATTERN”
The caller has passed pattern (a string).
The index search key will be
the characters in the pattern as far as the first magic character.
Lua’s magic characters are % ^ $ ( ) . [ ] * + - ?.
For example, if the pattern is “ABC.E”, the period is a magic
character and therefore the index search key will be “ABC”.
But there is a complication … If we see “%” followed by a punctuation
character, that punctuation character is “escaped” so
remove the “%” when making the index search key. For example, if the
pattern is “AB%$E”, the dollar sign is escaped and therefore
the index search key will be “AB$E”.
Finally there is a check that the index search key length
must be at least three – this is an arbitrary number, and in
fact zero would be okay, but short index search keys will cause
long search times.
NOTE #3 – “OUTER LOOP: INITIATE”
The function’s job is to return a result set,
just as box.space...select <box_space-select> would. We will fill
it within an outer loop that contains an inner
loop. The outer loop’s job is to execute the inner
loop, and possibly yield, until the search ends.
The inner loop’s job is to find tuples via the index, and put
them in the result set if they match the pattern.
NOTE #4 “INNER LOOP: ITERATOR”
The for loop here is using pairs(), see the
explanation of what index iterators are.
Within the inner loop,
there will be a local variable named “tuple” which contains
the latest tuple found via the index search key.
NOTE #5 “INNER LOOP: BREAK IF INDEX KEY IS TOO GREAT”
The iterator is GE (Greater or Equal), and we must be
more specific: if the search index key has N characters,
then the leftmost N characters of the result’s index field
must not be greater than the search index key. For example,
if the search index key is ‘ABC’, then ‘ABCDE’ is
a potential match, but ‘ABD’ is a signal that
no more matches are possible.
NOTE #6 “INNER LOOP: BREAK AFTER EVERY 10 TUPLES – MAYBE”
This chunk of code is for cooperative multitasking.
The number 10 is arbitrary, and usually a larger number would be okay.
The simple rule would be “after checking 10 tuples, yield,
and then resume the search (that is, do the inner loop again)
starting after the last value that was found”. However, if
the index is non-unique or if there is more than one field
in the index, then we might have duplicates – for example
{“ABC”,1}, {“ABC”, 2}, {“ABC”, 3}” – and it would be difficult
to decide which “ABC” tuple to resume with. Therefore, if
the result’s index field is the same as the previous
result’s index field, there is no break.
NOTE #7 “INNER LOOP: ADD TO RESULT SET IF PATTERN MATCHES”
Compare the result’s index field to the entire pattern.
For example, suppose that the caller passed pattern “ABC.E”
and there is an indexed field containing “ABCDE”.
Therefore the initial index search key is “ABC”.
Therefore a tuple containing an indexed field with “ABCDE”
will be found by the iterator, because “ABCDE” > “ABC”.
In that case string.match will return a value which is not nil.
Therefore this tuple can be added to the result set.
NOTE #8 “OUTER LOOP: BREAK, OR YIELD AND CONTINUE”
There are three conditions which will cause a break from
the inner loop: (1) the for loop ends naturally because
there are no more index keys which are greater than or
equal to the index search key, (2) the index key is too
great as described in NOTE #5, (3) it is time for a yield
as described in NOTE #6. If condition (1) or condition (2)
is true, then there is nothing more to do, the outer loop
ends too. If and only if condition (3) is true, the
outer loop must yield and then continue. If it does
continue, then the inner loop – the iterator search –
will happen again with a new value for the index search key.
EXAMPLE:
Start Tarantool, cut and paste the code for function indexed_pattern_search(),
and try the following:
box.space.t:drop()
box.schema.space.create('t')
box.space.t:create_index('primary',{})
box.space.t:create_index('secondary',{unique=false,parts={2,'string',3,'string'}})
box.space.t:insert{1,'A','a'}
box.space.t:insert{2,'AB',''}
box.space.t:insert{3,'ABC','a'}
box.space.t:insert{4,'ABCD',''}
box.space.t:insert{5,'ABCDE','a'}
box.space.t:insert{6,'ABCDE',''}
box.space.t:insert{7,'ABCDEF','a'}
box.space.t:insert{8,'ABCDF',''}
indexed_pattern_search("t", 2, "ABC.E.")
The result will be:
C tutorial
Tarantool can call C code with modules,
or with ffi,
or with C stored procedures.
This tutorial only is about the third option, C stored procedures.
In fact the routines are always “C functions” but the phrase
“stored procedure” is commonly used for historical reasons.
In this tutorial, which can be followed by anyone with a Tarantool
development package and a C compiler, there are five tasks:
- easy.c – prints “hello world”;
- harder.c – decodes a passed parameter value;
- hardest.c – uses the C API to do a DBMS insert;
- read.c – uses the C API to do a DBMS select;
- write.c – uses the C API to do a DBMS replace.
After following the instructions, and seeing that the results
are what is described here, users should feel confident about
writing their own stored procedures.
Check that these items exist on the computer:
- Tarantool 2.1 or later
- A gcc compiler, any modern version should work
module.h and files #included in itmsgpuck.hlibmsgpuck.a (only for some recent msgpuck versions)
The module.h file will exist if Tarantool was installed from source.
Otherwise Tarantool’s “developer” package must be installed.
For example on Ubuntu say:
$ sudo apt-get install tarantool-dev
or on Fedora say:
$ dnf -y install tarantool-devel
The msgpuck.h file will exist if Tarantool was installed from source.
Otherwise the “msgpuck” package must be installed from
https://github.com/tarantool/msgpuck.
Both module.h and msgpuck.h must be on the include path for the
C compiler to see them.
For example, if module.h address is /usr/local/include/tarantool/module.h,
and msgpuck.h address is /usr/local/include/msgpuck/msgpuck.h,
and they are not currently on the include path, say:
$ export CPATH=/usr/local/include/tarantool:/usr/local/include/msgpuck
The libmsgpuck.a static library is necessary with msgpuck versions
produced after February 2017. If and only if you encounter linking
problems when using the gcc statements in the examples for this tutorial, you should
put libmsgpuck.a on the path (libmsgpuck.a is produced from both msgpuck
and Tarantool source downloads so it should be easy to find). For
example, instead of “gcc -shared -o harder.so -fPIC harder.c”
for the second example below, you will need to say
“gcc -shared -o harder.so -fPIC harder.c libmsgpuck.a”.
Requests will be done using Tarantool as a
client.
Start Tarantool, and enter these requests.
box.cfg{listen=3306}
box.schema.space.create('capi_test')
box.space.capi_test:create_index('primary')
net_box = require('net.box')
capi_connection = net_box:new(3306)
In plainer language: create a space named capi_test,
and make a connection to self named capi_connection.
Leave the client running. It will be necessary to enter more requests later.
Start another shell. Change directory (cd) so that it is
the same as the directory that the client is running on.
Create a file. Name it easy.c. Put these six lines in it.
#include "module.h"
int easy(box_function_ctx_t *ctx, const char *args, const char *args_end)
{
printf("hello world\n");
return 0;
}
int easy2(box_function_ctx_t *ctx, const char *args, const char *args_end)
{
printf("hello world -- easy2\n");
return 0;
}
Compile the program, producing a library file named easy.so:
$ gcc -shared -o easy.so -fPIC easy.c
Now go back to the client and execute these requests:
box.schema.func.create('easy', {language = 'C'})
box.schema.user.grant('guest', 'execute', 'function', 'easy')
capi_connection:call('easy')
If these requests appear unfamiliar,
re-read the descriptions of
box.schema.func.create(),
box.schema.user.grant()
and conn:call().
The function that matters is capi_connection:call('easy').
Its first job is to find the ‘easy’ function, which should
be easy because by default Tarantool looks on the current
directory for a file named easy.so.
Its second job is to call the ‘easy’ function.
Since the easy() function in easy.c begins with printf("hello world\n"),
the words “hello world” will appear on the screen.
Its third job is to check that the call was successful.
Since the easy() function in easy.c ends with return 0,
there is no error message to display and the request is over.
The result should look like this:
Now let’s call the other function in easy.c – easy2().
This is almost the same as the easy() function, but there’s a detail:
when the file name is not the same as the function name,
then we have to specify
file-name.function-name.
box.schema.func.create('easy.easy2', {language = 'C'})
box.schema.user.grant('guest', 'execute', 'function', 'easy.easy2')
capi_connection:call('easy.easy2')
… and this time the result will be “hello world – easy2”.
Conclusion: calling a C function is easy.
Go back to the shell where the easy.c program was created.
Create a file. Name it harder.c. Put these 17 lines in it:
#include "module.h"
#include "msgpuck.h"
int harder(box_function_ctx_t *ctx, const char *args, const char *args_end)
{
uint32_t arg_count = mp_decode_array(&args);
printf("arg_count = %d\n", arg_count);
uint32_t field_count = mp_decode_array(&args);
printf("field_count = %d\n", field_count);
uint32_t val;
int i;
for (i = 0; i < field_count; ++i)
{
val = mp_decode_uint(&args);
printf("val=%d.\n", val);
}
return 0;
}
Compile the program, producing a library file named harder.so:
$ gcc -shared -o harder.so -fPIC harder.c
Now go back to the client and execute these requests:
box.schema.func.create('harder', {language = 'C'})
box.schema.user.grant('guest', 'execute', 'function', 'harder')
passable_table = {}
table.insert(passable_table, 1)
table.insert(passable_table, 2)
table.insert(passable_table, 3)
capi_connection:call('harder', {passable_table})
This time the call is passing a Lua table (passable_table)
to the harder() function. The harder() function will see it,
it’s in the char *args parameter.
At this point the harder() function will start using functions
defined in msgpuck.h.
The routines that begin with “mp” are msgpuck functions that
h
