Tarantool 2.3.1
Release: v. 2.3.1 Date: 2019-12-31 Tag: 2.3.1-0-g5a1a220
2.3.1 is the beta version of the 2.3 release series.
This release introduces roughly 38 features and resolves 102 bugs since the 2.2.1 version. There may be bugs in less common areas. If you find any, feel free to report an issue at GitHub.
Notable changes are:
- field name and JSON path updates
- anonymous replica
- new DOUBLE SQL type (and new ‘double’ box field type)
- stored and indexed decimals (and new ‘decimal’ field type)
fiber.top()
- feed data from a memory during replica initial join
- SQL prepared statements
- sessions settings service space
Aside of that many other features have been implemented and considerable amount of bugs have been fixed.
Tarantool 2.x is backward compatible with Tarantool 1.10.x in binary data layout, client-server protocol and replication protocol.
Please
upgrade
using the box.schema.upgrade()
procedure to unlock all the new
features of the 2.x series.
There are changes labeled with [Breaking change]. It means that the old behaviour was considered error-prone and therefore changed to protect users from unintended mistakes. However, there is a little probability that someone can lean on the old behaviour, and this label is to bring attention to the things that have been changed.
Introduce prepared statements support and prepared statements cache (gh-2592, gh-3292). Using of prepared statements allows to eliminate overhead of transmitting a statement text over a network and parsing it each time before execution. Aside of this, it allows to acquire binding parameters and result set columns metainformation prior to actual execution of a statement. This feature is vital for implementing standard DBMS APIs, such as ODBC and JDBC.
[Breaking change] Introduce _session_setting service space as replacement for PRAGMA keyword (gh-4511). All frontends (C, Lua, SQL, binary protocol) can use this space to access or update session settings. Removed
count_changes
,short_column_names
,sql_compound_select_limit
,vdbe_addoptrace
pragmas. Transformed others into _session_settings tuples.Extend SQL result set metadata (gh-4407), In addition to the
name
andtype
fields, thecollation
,is_nullable
,is_autoincrement
, andspan
fields are added. These new fields are shown when thefull_metadata
session setting is enabled but always sent via binary protocol.Add an ability to disable check constraints (gh-4244). Example:
ALTER TABLE foo {ENABLE|DISABLE} CHECK CONSTRAINT bar;
. For details of using from Lua, refer to documentation.AUTOINCREMENT for multipart primary key (gh-4217). The auto-increment feature can be set to any
INTEGER
orUNSIGNED
field ofPRIMARY KEY
using one of the two ways:AUTOINCREMENT
in column definition:CREATE TABLE t (i INT, a INT AUTOINCREMENT, PRIMARY KEY (i, a)); CREATE TABLE t (i INT AUTOINCREMENT, a INT, PRIMARY KEY (i, a));
AUTOINCREMENT
inPRIMARY KEY
definition:CREATE TABLE t (i INT, a INT, PRIMARY KEY (i, a AUTOINCREMENT)); CREATE TABLE t (i INT, a INT, PRIMARY KEY (i AUTOINCREMENT, a));
Allow to create a view from any CTE (common table expression) using
WITH
clause (gh-4149).Support user-defined functions in SQL.
box.schema.func.create()
API has been extended and should be used to make some function available in SQL. For details on fields added, refer to the description here: (doc-879). Usage of legacy mechanismbox.internal.sql_function_create
is forbidden now (gh-2200, gh-2233, gh-4113).Scalar functions
MIN/MAX
are renamed toLEAST/GREATEST
(gh-4405)Introduce
WITH ENGINE
clause forCREATE TABLE
statement (gh-4422). To allow a user to specify engine as per table option,CREATE TABLE
statement has been extended with optionalWITH ENGINE = <engine_name>
clause. This clause comes at the end ofCREATE TABLE
statement. For example:CREATE TABLE t_vinyl (id INT PRIMARY KEY) WITH ENGINE = 'vinyl';
Refer also to SQL reference documentation.
Introduce
DOUBLE
type (gh-3812).Display line and position in syntax errors (gh-2611).
Make constraint names unique within a table (gh-3503). The SQL standard requires
PRIMARY KEY
,UNIQUE
,FOREIGN KEY
andCHECK
constraints to have the unique name within a table. Now Tarantool/SQL follows this requirement. Please refer to (doc-1053).Optimization: a tuple already stores a map of offsets of indexed values. After the change, when a field after an indexed field is accessed, the tuple is decoded from the indexed field rather then from beginning (gh-4267).
[Breaking change] Drop
rows_per_wal
option ofbox.cfg()
in favor ofwal_max_size
(gh-3762).Decimals can now be stored in spaces. The corresponding field type is introduced:
decimal
. Decimal values are also allowed in thescalar
,any
, andnumber
fields. Decimal values can be indexed (gh-4333). Also refer to documentation onAdd support for decimals in update operations (gh-4413).
tuple:update()
and<space_object>:update()
now support decimal operands for arithmetic operations (‘+’ and ‘-’). The syntax is as usual, for example:tarantool> d = box.tuple.new(decimal.new('1')) --- ... tarantool> d:update{{'+', 1, decimal.new('0.5')}} --- - [1.5] ...
Insertion (‘!’) and assignment (‘=’) are also supported. See also the full description of the
update()
function in documentation.Allow to encode/decode decimals to
MsgPack
and to encode toYAML
andJSON
. Part of (gh-4333); 485439e3; documentation: (doc-992).Introduce field name and JSON path updates (gh-1261).
Example of update by a field name:
box.space.test:update({{'=', 'foo', 42}})
.JSON path update allows to change a value that is nested inside an array or a map. It provides convenient syntax (that is also available for connectors), consumes less space in WAL than replace, and is faster than replaces written in Lua. Example:
box.space.test:update({{'=', 'foo.bar[1]', 42}})
. Please refer to documentation here: (doc-1051).Introduce
double
field type. Part of (gh-3812). Though is not very usable in Lua, this new field type has been added inbox
as a base for the SQLDOUBLE
type.vinyl: don’t pin index for iterator lifetime (prerequisite for snapshot iterators). 02da82ea
vinyl: don’t exempt dropped indexes from dump and compaction (prerequisite for snapshot iterators). d7387ec9
box.info().replication
shows applier/replay’s latest error message. Now it also shows the errno description for system errors when it’s applicable (gh-4402).- Feed data from a memory during replica initial join (gh-1271). Aside of obvious speed up from reading from a memory instead of a disk, a read view that is acquired to perform an initial join may be a way more recent, that eliminates the need to play all xlog files since a last snapshot. Now relay need to send only changes that occur during initial join to finally join a replica.
- Introduce a new replica type - anonymous replica (gh-3186). Anonymous replica is not present in cluster space and so there is no limitation for its count in a replica set. Anonymous replica is read-only, but can be deanonymized and enabled for writes. Please refer to documentation: (doc-1050) for API and details.
- Expose
require('tarantool').package
which is ‘Tarantool’ for the community version and ‘Tarantool Enterprise’ for the enterprise version (gh-4408). This value is already displayed in a console greeting and inbox.info().package
, but it was not accessible from Lua before the firstbox.cfg{<...>}
call. - decimal: add modulo operator (
decimal.new(172.51) % 1 == 0.51
), part of (gh-4403). - [Breaking change] JSON and msgpack serializers now raise an error
when a depth of data nesting exceeds the
encode_max_depth
option value. The default value of the option has been increased from 32 to 128. Theencode_deep_as_nil
option is added to give an ability to set the old behaviour back (gh-4434). Notes:- These options can be set by using
json.cfg({<...>})
ormsgpack.cfg({<...>})
. box
data modification functions (insert
,replace
,update
andupsert
) follow the options of the default msgpack serializer instance, and now these functions raise an error on too many levels of nested data by default rather than cut the data silently. This behaviour can be configured usingmsgpack.cfg({<...>})
.- previously,
box.tuple.new()
,space:update()
,space:upsert()
and several other functions did not followencode_max_depth
option; now they do (see also the Bug fixed section). - previously,
json.cfg
andmsgpack.cfg
tables was not updated when an option had changed; now they show actual values (see also the Bug fixed section).
- These options can be set by using
- Show line and column in
json.decode()
errors (gh-3316). - Exit gracefully when a main script throws an error: notify systemd, log the error (gh-4382).
- key_def: accept both
field
andfieldno
inkey_def.new(<...>)
(gh-4519). Originallykey_def.new(<...>)
accepted onlyfieldno
to allow creation with<index_object>.parts
as argument. However, index definition format (<space_object>.create_index(<...>)
) is different and requiresfield
. Now both are supported. - Enable
__pairs
and__ipairs
metamethods from Lua 5.2 (gh-4560). We still conform Lua 5.1 API that is not always compatible with Lua 5.2. The change is only about those metamethods. - Implement a new function
fiber.top()
. It returns a table with all fibers alive and lists their CPU consumption. For details, refer to documentation. (gh-2694) - Expose
errno
field for box.error objects representing system errors. Part of (gh-4402).
- Modify type of a binding value in query response metainformation: always return INTEGER rather than UNSIGNED, even for positive values. This is necessary for consistency with integer literal types. b7d595ac.
- Reuse noSQL way to compare floating point values with integral ones.
This allows to handle corner cases like
SELECT 18446744073709551615.0 > 18446744073709551615
uniformly. 73a4a525. - Create or alter a table with a foreign key may lead to wrong bytecode generation that may cause a crash or wrong result (gh-4495).
- Allow to update a scalar value using SQL in a space that was created
from Lua and contains
array
,map
orany
fields (gh-4189). Note: Tarantool/SQL provides operations on scalar types and does not support ‘array’ and ‘map’ per se. - Allow nil to be returned from user-defined function (created with
box.schema.func.create()
). 1b39cbcf - Don’t drop a manually created sequence in DROP TABLE statement. a1155c8b
- Remove grants associated with the table in DROP TABLE statement (gh-4546).
- Fix segfault in
sql_expr_coll()
whenSUBSTR()
is called without arguments. 4c13972f - Fix converting of floating point values from range [2^63, 2^64] to integer (gh-4526).
- Make type string case lower everywhere: in error messages, meta
headers, and results of the
typeof()
SQL function. ee60d31d - Make the
LENGTH()
function to accept boolean argument (gh-4462). - Make implicit cast from BOOLEAN to TEXT to return uppercase for consistency with explicit cast (gh-4462).
- Fix segfault on binding a value as LIKE argument (gh-4566).
- For user-defined functions, verify that the returned value is of the type specified in the function definition (gh-4387).
- Start using comprehensive serializer
luaL_tofield()
to prepare LUA arguments for user-defined functions. This allows to support cdata types returned from Lua function (gh-4387). - An error is raised when a user-defined function returns too many values (gh-4387).
- Store a name of user-defined function in VDBE program instead of pointer. This allows to normally handle the situation when a user-defined function has been deleted to the moment of the VDBE code execution (gh-4176).
- Fix casting of VARBINARY value to a NUMBER (gh-4356)
- Print the data type instead of the data itself in
diag_set()
in case of binary data. The reason of this patch is thatLibYAML
converts the whole error message tobase64
in case of non-printable symbols. Part of (gh-4356). - Remove
ENGINE
from the list of the reserved keywords and allow to use it for identifiers: we are going to use the word as a name of some fields for tables forming informational schema. - Fix segfault when
LEAST()
orGREATEST()
built-in function is invoked without arguments (gh-4453). - Fix dirty memory access when constructing query plan involving search of floating point value in index over integer field (gh-4558).
INDEXED BY
clause now obligates the query planner to choose provided index. 49fedfe3
- Make functional index creation transactional (gh-4401)
- Detect a new invalid JSON path case (gh-4419)
- Randomize the next checkpoint time after manual
box.snapshot()
execution also (gh-4432). - Fix memory leak in call/eval in case of a transaction is not committed (gh-4388)
- Eliminate warning re
strip_core
option ofbox.cfg()
on MacOS and FreeBSD (gh-4464) - The msgpack serializer that is under
box.tuple.new()
(called tuple serializer) now reflects options set bymsgpack.cfg({<...>})
. Part of (gh-4434). Aside ofbox.tuple.new()
behaviour itself, it may affecttuple:frommap()
, methods of key_def Lua module, tuple and table merger sources, net.box results of:select()
and:execute()
calls, and xlog Lua module. box
functionsupdate
andupsert
now followmsgpack.cfg({encode_max_depth = <...>}
option. Part of (gh-4434).- fiber: make sure the guard page is created; refuse to create a new fiber otherwise (gh-4541). It is possible in case of heavy memory usage, say, when there is no resources to split VMAs.
- recovery: build secondary indices in the hot standby mode without waiting till the main instance termination (gh-4135).
- Fix error message for incorrect return value of functional index
extractor function (gh-4553).
- Was: “Key format doesn’t match one defined in functional index ‘’ of space ‘’: supplied key type is invalid: expected boolean”
- Now: “<…>: expected array”
- JSON path index now consider is_nullable property when a space had a format (gh-4520).
- Forbid
00000000-0000-0000-0000-000000000000
as the value ofbox.cfg({<...>})
options:replicaset_uuid
andinstance_uuid
(gh-4282). It did not work as expected: the nil UUID was treated as absence of the value. - Update cache of universe privileges without reconnect (gh-2763).
- net.box: fix memory leak in
net_box:connect(<URI>)
(gh-4588). - net.box: don’t fire the
on_connect
trigger on schema update (gh-4593). Also don’t fire theon_disconnect
trigger if a connection never entered into theactive
state (e.g. when the first schema fetch is failed). - func: fix use-after-free on function unload. fa2893ea
- Don’t destroy a session until
box.session.on_disconnect(<...>)
triggers are finished (gh-4627). This means, for example, thatbox.session.id()
can be safely invoked from theon_disconnect
trigger. Before this changebox.session.id()
returned garbage (usually 0) after yield in theon_disconnect
trigger. Note: tarantool/queue module is affected by this problem in some scenarios. It is especially suggested to update Tarantool at least to this release if you’re using this module. - func: Fix
box.schema.func.drop(<..>)
to unload unused modules (gh-4648). Also fixbox.schema.func.create(<..>)
to avoid loading a module again when another function from the module is loaded. - Encode Lua number -2^63 as integer in
msgpack.encode()
and box’s functions (gh-4672). - Forbid to drop admin’s universe access. 2de398ff. Bootstrap and recovery work on behalf of admin and should be able to fill in the system spaces. Drop of admin’s access may lead to an unrecoverable cluster.
- Refactor rope library to eliminate virtual calls to increase performance of the library (mainly for JSON path updates). baa4659c
- Refactor update operation code to avoid extra region-related arguments to take some performance boost (mainly for JSON path updates). dba9dba7
- Error logging has been removed in
engine_find()
to get rid of the error message duplication. 35177fe0. - decimal: Fix encoding of numbers with positive exponent. Follow-up (gh-692).
- Increment schema version on DDL operations where it did not performed before: alter of trigger, check constraint and foreign key constraint. Part of (gh-2592).
- Stop relay on subscribe error (gh-4399).
- Set
last_row_time
tonow
inrelay_new
andrelay_start
(gh-4431). - Do not abort replication on ER_UNKNOWN_REPLICA (gh-4455).
- Enter orphan mode on manual replication configuration change (gh-4424).
- Disallow bootstrap of read-only masters (gh-4321).
- Prefer to bootstrap a replica from a fully bootstrapped instance rather than from an instance that is in the process of bootstrapping (gh-4527). This change enables the case when two nodes (B, C) are being bootstrapped simultaneously using the one that is already bootstrapped (A), while A is configured to replicate from {B, C} and B – from {A, C}.
- Return immediately from
box.cfg{<...>}
when an instance is reconfigured withreplication_connect_quorum = 0
(gh-3760). This change also fixes the behaviour of reconfiguration with non-zeroreplication_connect_quorum
:box.cfg{<...>}
returns immediately regardless of whether connections to upstreams are established. - Apply replication settings of
box.cfg({<...>})
in a strict order (gh-4433). - Auto reconnect a replica if password is invalid (gh-4550).
box.session.su(<username>)
now correctly reports an error for<username>
longer thanBOX_NAME_MAX
which is 65000. 8b6bdb43- Was: ‘C++ exception’
- Now: ‘name length <…> is greater than BOX_NAME_MAX’
- Use empty password when a URI in
box.cfg{replication = <...>}
is likelogin@host:port
(gh-4605). The behaviour matches the net.box’s one now. Explicitlogin:@host:port
was necessary before, otherwise a replica displayed the following error: > Missing mandatory field ‘tuple’ in request - Fix segfault during replication configuration
(
box.cfg{replication = <...>}
call) (gh-4440, gh-4576, gh-4586, gh-4643). - Cancel a replica joining thread forcefully on Tarantool instance exit (gh-4528).
- Fix the applier to run the
<space>.before_replace
trigger during initial join (gh-4417).
- Fix segfault on
ffi.C._say()
without filename (gh-4336). - Fix
pwd.getpwall()
andpwd.getgrall()
hang on CentOS 6 and FreeBSD 12 (gh-4428, gh-4447). - json.encode() now follows
encode_max_depth
option for arrays that leads to a segfault on recursive Lua tables with numeric keys (gh-4366). - fio.mktree() now reports an error for existing non-directory file (gh-4439).
json.cfg
andmsgpack.cfg
tables were not updated when an option is changed. Part of (gh-4434).- Fix handling of a socket read error in the console client
(
console.connect(<URI>)
ortarantoolctl connect/enter <...>
). 89ec1d97 - Handle the “not enough memory” error gracefully when it is raised
from
lua_newthread()
(gh-4556). There are several cases when a new Lua thread is created:- Start executing a Lua function call or an eval request (from a
binary protocol, SQL or with
box.func.<...>:call()
). - Create of a new fiber.
- Start execution of a trigger.
- Start of encoding into a YAML format (
yaml.encode()
).
- Start executing a Lua function call or an eval request (from a
binary protocol, SQL or with
- Fix stack-use-after-scope in
json.decode()
(gh-4637). - Allow to register several functions using
box.schema.func.create()
, whose names are different only in letters case (gh-4561). This make function names work consistently with other names in tarantool (except SQL, of course). - Fix decimal comparison with nil. Follow-up (gh-692).
- Fix decimal comparison with
box.NULL
(gh-4454). - A pointer returned by
msgpack.decode*(cdata<[char] const *>)
functions can be assigned to buffer.rpos now (and the same for msgpackffi) (gh-3926). All those functions now returncdata<char *>
orcdata<const char *>
depending of a passed argument. Example of the code that did not work:res, buf.rpos = msgpack.decode(buf.rpos, buf:size())
. - lua/pickle: fix typo that leads to reject of negative integers for ‘i’ (integer) and ‘N’ (big-endian integer) formats in pickle.pack(). e2d9f664
- Use bundled
libcurl
rather than system-wide by default. (gh-4318, gh-4180, gh-4288, gh-4389, gh-4397). This closes several known problems that were fixed in recent libcurl versions, including segfaults, hangs, memory leaks and performance problems. - Fix assertion fail after a curl write error (gh-4232).
- Disable verbose mode when
{verbose = false}
is passed. 72613bb0
A new Lua output format is still in the alpha stage and has the known flaws, but we are working to make it rich and stable.
- Output
box.NULL
as"box.NULL"
rather than"cdata<void *>: NULL"
, part of (gh-3834) (in quotes for now, yes, due to (gh-4585) - Add semicolon (
;
) as responses delimiter (EOS, end of stream/statement), analogue of YAMLs end-of-document (...
) marker. This is vital for remote clients to determine the end of a particular response, part of (gh-3834). - Fix hang in the console client (
console.connect(<URI>)
ortarantoolctl connect/enter <...>
) after\set output lua[,block]
command, part of (gh-3834). In order to overcome it, two changes have been made:- Parse
\set output lua[,block]
command on a client prior to sending it to a server, store current responses delimiter (EOS) and use it to determine end of responses. - Send
\set output <...>
command with a default output mode when establishing a connection (it is matter if different default modes are set).
- Parse
- Provide an ability to get or set current responses delimiter using
console.eos([<...>])
, part of (gh-3834).
- Fix fold machinery misbehaves (gh-4376).
- Fix for
debug.getinfo(1,'>S')
(gh-3833). - Fix
string.find
recording (gh-4476). - Fix the “Data segment size exceeds process limit” error on FreeBSD/x64: do not change resource limits when it is not necessary (gh-4537).
- fold: keep type of emitted CONV in sync with its mode. LuaJIT#524 This fixes the following assertion fail: > asm_conv: Assertion `((IRType)((ir->t).irt & IRT_TYPE)) != st’ failed
- Support
systemd
’s NOTIFY_SOCKET on OS X (gh-4436). - Fix linking with static
openssl
library (gh-4437). - Get rid of warning re empty
NOTIFY_SOCKET
variable (gh-4305). - rocks: fix ‘invalid date format’ error when installing a packed rock (gh-4481).
- Remove libyaml from rpm/deb dependencies, because we use bunbled version of libyaml for the packages (since 2.2.1) (gh-4442).
- Fix CLI boolean options handling in
tarantoolctl cat <...>
, such as--show-system
(gh-4076). - Fix segfault (out of bounds access) when a stack unwinding error
occurs at backtrace printing (gh-4636). Backtrace is printed on the
SIGFPE and SIGSEGV signals or when LuaJIT finds itself in the
unrecoverable state (
lua_atpanic()
). - Clear terminal state on panic (gh-4466).
- access: fix the invalid error type
box.session.su()
raises for a not found user- was:
SystemError
- now:
ClientError
- was:
- Fix for GCC 4.8.5, which is default version on CentOS 7 (gh-4438).
- Fix OpenSSL linking problems on FreeBSD (gh-4490).
- Fix linking problems on Mac OS when several toolchains are in PATH (gh-4587).
- Fix GCC 9 warning on strncpy() (gh-4515).
- Fix build on Mac with gcc and XCode 11 (gh-4580).
- Fix LTO warnings that were treated as errors in a release build (gh-4512).