Tarantool 2.3.1
Release: v. 2.3.1 Date: 2019-12-31 Tag: 2.3.1-0-g5a1a220
2.3.1 is the beta version of the 2.3 release series.
This release introduces roughly 38 features and resolves 102 bugs since the 2.2.1 version. There may be bugs in less common areas. If you find any, feel free to report an issue at GitHub.
Notable changes are:
- field name and JSON path updates
- anonymous replica
- new DOUBLE SQL type (and new ‘double’ box field type)
- stored and indexed decimals (and new ‘decimal’ field type)
fiber.top()- feed data from a memory during replica initial join
- SQL prepared statements
- sessions settings service space
Aside of that many other features have been implemented and considerable amount of bugs have been fixed.
Tarantool 2.x is backward compatible with Tarantool 1.10.x in binary data layout, client-server protocol and replication protocol.
Please
upgrade
using the box.schema.upgrade() procedure to unlock all the new
features of the 2.x series.
There are changes labeled with [Breaking change]. It means that the old behaviour was considered error-prone and therefore changed to protect users from unintended mistakes. However, there is a little probability that someone can lean on the old behaviour, and this label is to bring attention to the things that have been changed.
Introduce prepared statements support and prepared statements cache (gh-2592, gh-3292). Using of prepared statements allows to eliminate overhead of transmitting a statement text over a network and parsing it each time before execution. Aside of this, it allows to acquire binding parameters and result set columns metainformation prior to actual execution of a statement. This feature is vital for implementing standard DBMS APIs, such as ODBC and JDBC.
[Breaking change] Introduce _session_setting service space as replacement for PRAGMA keyword (gh-4511). All frontends (C, Lua, SQL, binary protocol) can use this space to access or update session settings. Removed
count_changes,short_column_names,sql_compound_select_limit,vdbe_addoptracepragmas. Transformed others into _session_settings tuples.Extend SQL result set metadata (gh-4407), In addition to the
nameandtypefields, thecollation,is_nullable,is_autoincrement, andspanfields are added. These new fields are shown when thefull_metadatasession setting is enabled but always sent via binary protocol.Add an ability to disable check constraints (gh-4244). Example:
ALTER TABLE foo {ENABLE|DISABLE} CHECK CONSTRAINT bar;. For details of using from Lua, refer to documentation.AUTOINCREMENT for multipart primary key (gh-4217). The auto-increment feature can be set to any
INTEGERorUNSIGNEDfield ofPRIMARY KEYusing one of the two ways:AUTOINCREMENTin column definition:CREATE TABLE t (i INT, a INT AUTOINCREMENT, PRIMARY KEY (i, a)); CREATE TABLE t (i INT AUTOINCREMENT, a INT, PRIMARY KEY (i, a));
AUTOINCREMENTinPRIMARY KEYdefinition:CREATE TABLE t (i INT, a INT, PRIMARY KEY (i, a AUTOINCREMENT)); CREATE TABLE t (i INT, a INT, PRIMARY KEY (i AUTOINCREMENT, a));
Allow to create a view from any CTE (common table expression) using
WITHclause (gh-4149).Support user-defined functions in SQL.
box.schema.func.create()API has been extended and should be used to make some function available in SQL. For details on fields added, refer to the description here: (doc-879). Usage of legacy mechanismbox.internal.sql_function_createis forbidden now (gh-2200, gh-2233, gh-4113).Scalar functions
MIN/MAXare renamed toLEAST/GREATEST(gh-4405)Introduce
WITH ENGINEclause forCREATE TABLEstatement (gh-4422). To allow a user to specify engine as per table option,CREATE TABLEstatement has been extended with optionalWITH ENGINE = <engine_name>clause. This clause comes at the end ofCREATE TABLEstatement. For example:CREATE TABLE t_vinyl (id INT PRIMARY KEY) WITH ENGINE = 'vinyl';
Refer also to SQL reference documentation.
Introduce
DOUBLEtype (gh-3812).Display line and position in syntax errors (gh-2611).
Make constraint names unique within a table (gh-3503). The SQL standard requires
PRIMARY KEY,UNIQUE,FOREIGN KEYandCHECKconstraints to have the unique name within a table. Now Tarantool/SQL follows this requirement. Please refer to (doc-1053).Optimization: a tuple already stores a map of offsets of indexed values. After the change, when a field after an indexed field is accessed, the tuple is decoded from the indexed field rather then from beginning (gh-4267).
[Breaking change] Drop
rows_per_waloption ofbox.cfg()in favor ofwal_max_size(gh-3762).Decimals can now be stored in spaces. The corresponding field type is introduced:
decimal. Decimal values are also allowed in thescalar,any, andnumberfields. Decimal values can be indexed (gh-4333). Also refer to documentation onAdd support for decimals in update operations (gh-4413).
tuple:update()and<space_object>:update()now support decimal operands for arithmetic operations (‘+’ and ‘-’). The syntax is as usual, for example:tarantool> d = box.tuple.new(decimal.new('1')) --- ... tarantool> d:update{{'+', 1, decimal.new('0.5')}} --- - [1.5] ...
Insertion (‘!’) and assignment (‘=’) are also supported. See also the full description of the
update()function in documentation.Allow to encode/decode decimals to
MsgPackand to encode toYAMLandJSON. Part of (gh-4333); 485439e3; documentation: (doc-992).Introduce field name and JSON path updates (gh-1261).
Example of update by a field name:
box.space.test:update({{'=', 'foo', 42}}).JSON path update allows to change a value that is nested inside an array or a map. It provides convenient syntax (that is also available for connectors), consumes less space in WAL than replace, and is faster than replaces written in Lua. Example:
box.space.test:update({{'=', 'foo.bar[1]', 42}}). Please refer to documentation here: (doc-1051).Introduce
doublefield type. Part of (gh-3812). Though is not very usable in Lua, this new field type has been added inboxas a base for the SQLDOUBLEtype.vinyl: don’t pin index for iterator lifetime (prerequisite for snapshot iterators). 02da82ea
vinyl: don’t exempt dropped indexes from dump and compaction (prerequisite for snapshot iterators). d7387ec9
box.info().replicationshows applier/replay’s latest error message. Now it also shows the errno description for system errors when it’s applicable (gh-4402).- Feed data from a memory during replica initial join (gh-1271). Aside of obvious speed up from reading from a memory instead of a disk, a read view that is acquired to perform an initial join may be a way more recent, that eliminates the need to play all xlog files since a last snapshot. Now relay need to send only changes that occur during initial join to finally join a replica.
- Introduce a new replica type - anonymous replica (gh-3186). Anonymous replica is not present in cluster space and so there is no limitation for its count in a replica set. Anonymous replica is read-only, but can be deanonymized and enabled for writes. Please refer to documentation: (doc-1050) for API and details.
- Expose
require('tarantool').packagewhich is ‘Tarantool’ for the community version and ‘Tarantool Enterprise’ for the enterprise version (gh-4408). This value is already displayed in a console greeting and inbox.info().package, but it was not accessible from Lua before the firstbox.cfg{<...>}call. - decimal: add modulo operator (
decimal.new(172.51) % 1 == 0.51), part of (gh-4403). - [Breaking change] JSON and msgpack serializers now raise an error
when a depth of data nesting exceeds the
encode_max_depthoption value. The default value of the option has been increased from 32 to 128. Theencode_deep_as_niloption is added to give an ability to set the old behaviour back (gh-4434). Notes:- These options can be set by using
json.cfg({<...>})ormsgpack.cfg({<...>}). boxdata modification functions (insert,replace,updateandupsert) follow the options of the default msgpack serializer instance, and now these functions raise an error on too many levels of nested data by default rather than cut the data silently. This behaviour can be configured usingmsgpack.cfg({<...>}).- previously,
box.tuple.new(),space:update(),space:upsert()and several other functions did not followencode_max_depthoption; now they do (see also the Bug fixed section). - previously,
json.cfgandmsgpack.cfgtables was not updated when an option had changed; now they show actual values (see also the Bug fixed section).
- These options can be set by using
- Show line and column in
json.decode()errors (gh-3316). - Exit gracefully when a main script throws an error: notify systemd, log the error (gh-4382).
- key_def: accept both
fieldandfieldnoinkey_def.new(<...>)(gh-4519). Originallykey_def.new(<...>)accepted onlyfieldnoto allow creation with<index_object>.partsas argument. However, index definition format (<space_object>.create_index(<...>)) is different and requiresfield. Now both are supported. - Enable
__pairsand__ipairsmetamethods from Lua 5.2 (gh-4560). We still conform Lua 5.1 API that is not always compatible with Lua 5.2. The change is only about those metamethods. - Implement a new function
fiber.top(). It returns a table with all fibers alive and lists their CPU consumption. For details, refer to documentation. (gh-2694) - Expose
errnofield for box.error objects representing system errors. Part of (gh-4402).
- Modify type of a binding value in query response metainformation: always return INTEGER rather than UNSIGNED, even for positive values. This is necessary for consistency with integer literal types. b7d595ac.
- Reuse noSQL way to compare floating point values with integral ones.
This allows to handle corner cases like
SELECT 18446744073709551615.0 > 18446744073709551615uniformly. 73a4a525. - Create or alter a table with a foreign key may lead to wrong bytecode generation that may cause a crash or wrong result (gh-4495).
- Allow to update a scalar value using SQL in a space that was created
from Lua and contains
array,maporanyfields (gh-4189). Note: Tarantool/SQL provides operations on scalar types and does not support ‘array’ and ‘map’ per se. - Allow nil to be returned from user-defined function (created with
box.schema.func.create()). 1b39cbcf - Don’t drop a manually created sequence in DROP TABLE statement. a1155c8b
- Remove grants associated with the table in DROP TABLE statement (gh-4546).
- Fix segfault in
sql_expr_coll()whenSUBSTR()is called without arguments. 4c13972f - Fix converting of floating point values from range [2^63, 2^64] to integer (gh-4526).
- Make type string case lower everywhere: in error messages, meta
headers, and results of the
typeof()SQL function. ee60d31d - Make the
LENGTH()function to accept boolean argument (gh-4462). - Make implicit cast from BOOLEAN to TEXT to return uppercase for consistency with explicit cast (gh-4462).
- Fix segfault on binding a value as LIKE argument (gh-4566).
- For user-defined functions, verify that the returned value is of the type specified in the function definition (gh-4387).
- Start using comprehensive serializer
luaL_tofield()to prepare LUA arguments for user-defined functions. This allows to support cdata types returned from Lua function (gh-4387). - An error is raised when a user-defined function returns too many values (gh-4387).
- Store a name of user-defined function in VDBE program instead of pointer. This allows to normally handle the situation when a user-defined function has been deleted to the moment of the VDBE code execution (gh-4176).
- Fix casting of VARBINARY value to a NUMBER (gh-4356)
- Print the data type instead of the data itself in
diag_set()in case of binary data. The reason of this patch is thatLibYAMLconverts the whole error message tobase64in case of non-printable symbols. Part of (gh-4356). - Remove
ENGINEfrom the list of the reserved keywords and allow to use it for identifiers: we are going to use the word as a name of some fields for tables forming informational schema. - Fix segfault when
LEAST()orGREATEST()built-in function is invoked without arguments (gh-4453). - Fix dirty memory access when constructing query plan involving search of floating point value in index over integer field (gh-4558).
INDEXED BYclause now obligates the query planner to choose provided index. 49fedfe3
- Make functional index creation transactional (gh-4401)
- Detect a new invalid JSON path case (gh-4419)
- Randomize the next checkpoint time after manual
box.snapshot()execution also (gh-4432). - Fix memory leak in call/eval in case of a transaction is not committed (gh-4388)
- Eliminate warning re
strip_coreoption ofbox.cfg()on MacOS and FreeBSD (gh-4464) - The msgpack serializer that is under
box.tuple.new()(called tuple serializer) now reflects options set bymsgpack.cfg({<...>}). Part of (gh-4434). Aside ofbox.tuple.new()behaviour itself, it may affecttuple:frommap(), methods of key_def Lua module, tuple and table merger sources, net.box results of:select()and:execute()calls, and xlog Lua module. boxfunctionsupdateandupsertnow followmsgpack.cfg({encode_max_depth = <...>}option. Part of (gh-4434).- fiber: make sure the guard page is created; refuse to create a new fiber otherwise (gh-4541). It is possible in case of heavy memory usage, say, when there is no resources to split VMAs.
- recovery: build secondary indices in the hot standby mode without waiting till the main instance termination (gh-4135).
- Fix error message for incorrect return value of functional index
extractor function (gh-4553).
- Was: “Key format doesn’t match one defined in functional index ‘’ of space ‘’: supplied key type is invalid: expected boolean”
- Now: “<…>: expected array”
- JSON path index now consider is_nullable property when a space had a format (gh-4520).
- Forbid
00000000-0000-0000-0000-000000000000as the value ofbox.cfg({<...>})options:replicaset_uuidandinstance_uuid(gh-4282). It did not work as expected: the nil UUID was treated as absence of the value. - Update cache of universe privileges without reconnect (gh-2763).
- net.box: fix memory leak in
net_box:connect(<URI>)(gh-4588). - net.box: don’t fire the
on_connecttrigger on schema update (gh-4593). Also don’t fire theon_disconnecttrigger if a connection never entered into theactivestate (e.g. when the first schema fetch is failed). - func: fix use-after-free on function unload. fa2893ea
- Don’t destroy a session until
box.session.on_disconnect(<...>)triggers are finished (gh-4627). This means, for example, thatbox.session.id()can be safely invoked from theon_disconnecttrigger. Before this changebox.session.id()returned garbage (usually 0) after yield in theon_disconnecttrigger. Note: tarantool/queue module is affected by this problem in some scenarios. It is especially suggested to update Tarantool at least to this release if you’re using this module. - func: Fix
box.schema.func.drop(<..>)to unload unused modules (gh-4648). Also fixbox.schema.func.create(<..>)to avoid loading a module again when another function from the module is loaded. - Encode Lua number -2^63 as integer in
msgpack.encode()and box’s functions (gh-4672). - Forbid to drop admin’s universe access. 2de398ff. Bootstrap and recovery work on behalf of admin and should be able to fill in the system spaces. Drop of admin’s access may lead to an unrecoverable cluster.
- Refactor rope library to eliminate virtual calls to increase performance of the library (mainly for JSON path updates). baa4659c
- Refactor update operation code to avoid extra region-related arguments to take some performance boost (mainly for JSON path updates). dba9dba7
- Error logging has been removed in
engine_find()to get rid of the error message duplication. 35177fe0. - decimal: Fix encoding of numbers with positive exponent. Follow-up (gh-692).
- Increment schema version on DDL operations where it did not performed before: alter of trigger, check constraint and foreign key constraint. Part of (gh-2592).
- Stop relay on subscribe error (gh-4399).
- Set
last_row_timetonowinrelay_newandrelay_start(gh-4431). - Do not abort replication on ER_UNKNOWN_REPLICA (gh-4455).
- Enter orphan mode on manual replication configuration change (gh-4424).
- Disallow bootstrap of read-only masters (gh-4321).
- Prefer to bootstrap a replica from a fully bootstrapped instance rather than from an instance that is in the process of bootstrapping (gh-4527). This change enables the case when two nodes (B, C) are being bootstrapped simultaneously using the one that is already bootstrapped (A), while A is configured to replicate from {B, C} and B – from {A, C}.
- Return immediately from
box.cfg{<...>}when an instance is reconfigured withreplication_connect_quorum = 0(gh-3760). This change also fixes the behaviour of reconfiguration with non-zeroreplication_connect_quorum:box.cfg{<...>}returns immediately regardless of whether connections to upstreams are established. - Apply replication settings of
box.cfg({<...>})in a strict order (gh-4433). - Auto reconnect a replica if password is invalid (gh-4550).
box.session.su(<username>)now correctly reports an error for<username>longer thanBOX_NAME_MAXwhich is 65000. 8b6bdb43- Was: ‘C++ exception’
- Now: ‘name length <…> is greater than BOX_NAME_MAX’
- Use empty password when a URI in
box.cfg{replication = <...>}is likelogin@host:port(gh-4605). The behaviour matches the net.box’s one now. Explicitlogin:@host:portwas necessary before, otherwise a replica displayed the following error: > Missing mandatory field ‘tuple’ in request - Fix segfault during replication configuration
(
box.cfg{replication = <...>}call) (gh-4440, gh-4576, gh-4586, gh-4643). - Cancel a replica joining thread forcefully on Tarantool instance exit (gh-4528).
- Fix the applier to run the
<space>.before_replacetrigger during initial join (gh-4417).
- Fix segfault on
ffi.C._say()without filename (gh-4336). - Fix
pwd.getpwall()andpwd.getgrall()hang on CentOS 6 and FreeBSD 12 (gh-4428, gh-4447). - json.encode() now follows
encode_max_depthoption for arrays that leads to a segfault on recursive Lua tables with numeric keys (gh-4366). - fio.mktree() now reports an error for existing non-directory file (gh-4439).
json.cfgandmsgpack.cfgtables were not updated when an option is changed. Part of (gh-4434).- Fix handling of a socket read error in the console client
(
console.connect(<URI>)ortarantoolctl connect/enter <...>). 89ec1d97 - Handle the “not enough memory” error gracefully when it is raised
from
lua_newthread()(gh-4556). There are several cases when a new Lua thread is created:- Start executing a Lua function call or an eval request (from a
binary protocol, SQL or with
box.func.<...>:call()). - Create of a new fiber.
- Start execution of a trigger.
- Start of encoding into a YAML format (
yaml.encode()).
- Start executing a Lua function call or an eval request (from a
binary protocol, SQL or with
- Fix stack-use-after-scope in
json.decode()(gh-4637). - Allow to register several functions using
box.schema.func.create(), whose names are different only in letters case (gh-4561). This make function names work consistently with other names in tarantool (except SQL, of course). - Fix decimal comparison with nil. Follow-up (gh-692).
- Fix decimal comparison with
box.NULL(gh-4454). - A pointer returned by
msgpack.decode*(cdata<[char] const *>)functions can be assigned to buffer.rpos now (and the same for msgpackffi) (gh-3926). All those functions now returncdata<char *>orcdata<const char *>depending of a passed argument. Example of the code that did not work:res, buf.rpos = msgpack.decode(buf.rpos, buf:size()). - lua/pickle: fix typo that leads to reject of negative integers for ‘i’ (integer) and ‘N’ (big-endian integer) formats in pickle.pack(). e2d9f664
- Use bundled
libcurlrather than system-wide by default. (gh-4318, gh-4180, gh-4288, gh-4389, gh-4397). This closes several known problems that were fixed in recent libcurl versions, including segfaults, hangs, memory leaks and performance problems. - Fix assertion fail after a curl write error (gh-4232).
- Disable verbose mode when
{verbose = false}is passed. 72613bb0
A new Lua output format is still in the alpha stage and has the known flaws, but we are working to make it rich and stable.
- Output
box.NULLas"box.NULL"rather than"cdata<void *>: NULL", part of (gh-3834) (in quotes for now, yes, due to (gh-4585) - Add semicolon (
;) as responses delimiter (EOS, end of stream/statement), analogue of YAMLs end-of-document (...) marker. This is vital for remote clients to determine the end of a particular response, part of (gh-3834). - Fix hang in the console client (
console.connect(<URI>)ortarantoolctl connect/enter <...>) after\set output lua[,block]command, part of (gh-3834). In order to overcome it, two changes have been made:- Parse
\set output lua[,block]command on a client prior to sending it to a server, store current responses delimiter (EOS) and use it to determine end of responses. - Send
\set output <...>command with a default output mode when establishing a connection (it is matter if different default modes are set).
- Parse
- Provide an ability to get or set current responses delimiter using
console.eos([<...>]), part of (gh-3834).
- Fix fold machinery misbehaves (gh-4376).
- Fix for
debug.getinfo(1,'>S')(gh-3833). - Fix
string.findrecording (gh-4476). - Fix the “Data segment size exceeds process limit” error on FreeBSD/x64: do not change resource limits when it is not necessary (gh-4537).
- fold: keep type of emitted CONV in sync with its mode. LuaJIT#524 This fixes the following assertion fail: > asm_conv: Assertion `((IRType)((ir->t).irt & IRT_TYPE)) != st’ failed
- Support
systemd’s NOTIFY_SOCKET on OS X (gh-4436). - Fix linking with static
openssllibrary (gh-4437). - Get rid of warning re empty
NOTIFY_SOCKETvariable (gh-4305). - rocks: fix ‘invalid date format’ error when installing a packed rock (gh-4481).
- Remove libyaml from rpm/deb dependencies, because we use bunbled version of libyaml for the packages (since 2.2.1) (gh-4442).
- Fix CLI boolean options handling in
tarantoolctl cat <...>, such as--show-system(gh-4076). - Fix segfault (out of bounds access) when a stack unwinding error
occurs at backtrace printing (gh-4636). Backtrace is printed on the
SIGFPE and SIGSEGV signals or when LuaJIT finds itself in the
unrecoverable state (
lua_atpanic()). - Clear terminal state on panic (gh-4466).
- access: fix the invalid error type
box.session.su()raises for a not found user- was:
SystemError - now:
ClientError
- was:
- Fix for GCC 4.8.5, which is default version on CentOS 7 (gh-4438).
- Fix OpenSSL linking problems on FreeBSD (gh-4490).
- Fix linking problems on Mac OS when several toolchains are in PATH (gh-4587).
- Fix GCC 9 warning on strncpy() (gh-4515).
- Fix build on Mac with gcc and XCode 11 (gh-4580).
- Fix LTO warnings that were treated as errors in a release build (gh-4512).