Tarantool 2.10.0-rc1

Field type UUID is now a part of field type SCALAR (gh-6042).

Field type UUID is now available in SQL, and a new UUID can be generated using the new SQL built-in uuid() function (gh-5886).

[Breaking change] timeout() method of net.box connection, which was marked deprecated more than four years ago (in 1.7.4), was dropped, because it negatively affected performance of hot net.box methods, like call() and select(), in case those were called without specifying a timeout (gh-6242).

Improved net.box performance by up to 70% by rewriting hot code paths in C (gh-6241).

Introduced compact tuples that allow saving 4 bytes per tuple in case of small userdata (gh-5385).

Streams and interactive transactions over streams are implemented in iproto. A stream is associated with its ID, which is unique within one connection. All requests with the same not zero stream ID belong to the same stream. All requests in the stream are processed synchronously. The execution of the next request will not start until the previous one is completed. If a request has zero stream ID, it does not belong to a stream and is processed in the old way.

In net.box, a stream is an object above connection that has the same methods but allows to execute requests sequentially. ID is generated automatically on the client side. If users write their own connector and want to use streams, they must transmit stream_id over iproto protocol. The main purpose of streams is transactions via iproto. Each stream can start its own transaction, allowing multiplexing several transactions over one connection. There are multiple ways to begin, commit and rollback a transaction: using appropriate stream methods, using call or eval methods or using execute method with SQL transaction syntax. Users can mix these methods, for example, start a transaction using stream:begin(), and commit a transaction using stream:call('box.commit') or stream:execute('COMMIT').

If any request fails during the transaction, it will not affect the other requests in the transaction. If a disconnect occurs when there is some active transaction in the stream, this transaction will be rolled back if it does not have time to commit before this moment.

Added new memtx_allocator option to box.cfg{} that allows to select the appropriate allocator for memtx tuples if necessary. Possible values are system for malloc allocator and small for default small allocator.

Implemented system allocator, based on malloc: slab allocator that is used for tuples allocation has a certain disadvantage—it tends to unresolvable fragmentation on certain workloads (size migration). In this case, a user should be able to choose another allocator. System allocator based on malloc function, but restricted by the same quota as slab allocator. System allocator does not alloc all memory at start. Instead, it allocates memory as needed, checking that quota is not exceeded (gh-5419).

Added box.stat.net.thread() for reporting per thread net statistics (gh-6293).

Added the new STREAMS metric to box.stat.net that contains statistics for iproto streams. STREAMS contains same counters as CONNECTIONS metric in box.stat.net: current, rps, and total (gh-6293).

Extended the network protocol (IPROTO) with a new request type (IPROTO_ID) that is supposed to be used for exchanging sets of supported features between server and client (gh-6253).

Added required_protocol_version and required_protocol_features to net.box connection options. The new options allow to specify the IPROTO protocol version and features that must be supported by the server for the connection to pass (gh-6253).

[Breaking change] Added msgpack.cfg.encode_error_as_ext configuration option to enable/disable encoding errors as MP_ERROR MsgPack extension. The option is enabled by default (gh-6433).

[Breaking change] Removed box.session.setting.error_marshaling_enabled. Error marshalling is now enabled automatically if the connector supports it (gh-6428).

Added new metrics REQUESTS_IN_PROGRESS and REQUESTS_IN_STREAM_QUEUE to box.stat.net that contains detailed statistics for iproto requests. These metrics contain same counters as other metrics in box.stat.net: current, rps, and total (gh-6293).

Implemented a timeout for fiber:join in Lua (gh-6203).

Updated libev to version 4.33 (gh-4909).

Added the new box.txn_id() function that returns the id of the current transaction if called within a transaction, nil otherwise.

Previously, if a yield occurs for a transaction that does not support it, we roll back all its statements but still process its new statements (they will roll back with each yield). Also, the transaction will be rolled back when a commit is attempted. Now we stop processing any new statements right after the first yield if a transaction does not support it.

Implemented a timeout for transactions after which they are rolled back (gh-6177).

Implemented new C API function box_txn_set_timeout to set a timeout for transactions.

Implemented a timeout for iproto transactions after which they are rolled back (gh-6177).

Implemented new IPROTO_TIMEOUT 0x56 key, which is used to set a timeout for transactions over iproto streams. It is stored in the body of IPROTO_BEGIN request.

Introduced box.broadcast and box.watch functions to signal/watch user-defined state changes (gh-6257).

Added watchers support to the network protocol (gh-6257).

Added watchers support to the net.box connector (gh-6257).

Error objects with the code box.error.READONLY now have additional fields explaining why the error happened.

Also, there is a new field box.info.ro_reason. It is nil on a writable instance, but reports a reason when box.info.ro is true (gh-5568).

Implemented the ability to open several listening sockets. In addition to the ability to pass uri as a number or string, added the ability to pass uri as a table of numbers or strings (gh-3554).

[Breaking change] net.box console support, which was marked deprecated in 1.10, was dropped. Use require('console').connect() instead.

Added takes_raw_args Lua function option for wrapping arguments in msgpack.object to skip decoding (gh-3349).

Implemented the graceful shutdown protocol for IPROTO connections (gh-5924).

Added fetch_schema flag to netbox.connect to control schema fetching from remote instance (gh-4789).

Added linking type (dynamic or static) to Tarantool build info.

Changed log level of some information messages from critical to info (gh-4675).

Added predefined system events: box.status, box.id, box.election, and box.schema (gh-6260).

Introduced transaction isolation levels in Lua and IPROTO (gh-6930).

Added support for backtrace feature on AARCH64 architecture (gh-6060).

Implemented collection of parent backtrace for the newly created fibers. To enable the feature, call fiber.parent_backtrace_enable. To disable it, call fiber.parent_backtrace_disable: disabled by default (gh-4302).

Introduced support for LJ_DUALNUM mode in luajit-gdb.py (gh-6224).

Introduced preliminary support of GNU/Linux ARM64 and MacOS M1. In the scope of this activity, the following issues have been resolved:

• Introduced support for a full 64-bit range of lightuserdata values (gh-2712).
• Fixed memory remapping issue when the page leaves 47-bit segments.
• Fixed M1 architecture detection (gh-6065).
• Fixed variadic arguments handling in FFI on M1 (gh-6066).
• Fixed table.move misbehavior when table reallocation occurs (gh-6084).
• Fixed Lua stack inconsistency when xpcall is called with an invalid second argument on ARM64 (gh-6093).
• Fixed BC_USETS bytecode semantics for closed upvalues and gray strings.
• Fixed side exit jump target patching considering the range values of the particular instruction (gh-6098).
• Fixed current Lua coroutine restoring on an exceptional path on ARM64 (gh-6189).

Now, memory profiler records allocations from traces grouping them by the trace number (gh-5814). The memory profiler parser can display the new type of allocation sources in the following format:

| TRACE [<trace-no>] <trace-addr> started at @<sym-chunk>:<sym-line>

Now the memory profiler reports allocations made by the JIT engine while compiling the trace as INTERNAL (gh-5679).

Now the memory profiler emits events of the new type when a function or a trace is created. As a result the memory profiler parser can enrich its symbol table with the new functions and traces (gh-5815).

Furthermore, there are symbol generations introduced within the internal parser structure to handle possible collisions of function addresses and trace numbers.

Now the memory profiler dumps symbol table for C functions. As a result memory profiler parser can enrich its symbol table with C symbols (gh-5813). Furthermore, now memory profiler dumps special events for symbol table when it encounters a new C symbol, that has not been dumped yet.

Introduced the LuaJIT platform profiler (gh-781) and the profile parser. This profiler is able to capture both host and VM stacks, so it can show the whole picture. Both C and Lua API’s are available for the profiler. Profiler comes with the default parser, which produces output in a flamegraph.pl-suitable format. The following profiling modes are available:

• Default: only virtual machine state counters.
• Leaf: shows the last frame on the stack.
• Callchain: performs a complete stack dump.

Introduced the new method table.equals. It compares two tables by value with respect to the __eq metamethod.

Added support of console autocompletion for net.box objects stream and future (gh-6305).

Added box.runtime.info().tuple metric to track the amount of memory occupied by tuples allocated on runtime arena (gh-5872).

It does not count tuples that arrive from memtx or vinyl but counts tuples created on-the-fly: say, using box.tuple.new(<...>).

[Breaking change] fiber.wakeup() in Lua and fiber_wakeup() in C became NOP on the currently running fiber.

Previously they allowed ignoring the next yield or sleep, which resulted in unexpected erroneous wake-ups. Calling these functions right before fiber.create() in Lua or fiber_start() in C could lead to a crash (in debug build) or undefined behaviour (in release build) (gh-6043).

There was a single use case for that—reschedule in the same event loop iteration which is not the same as fiber.sleep(0) in Lua and fiber_sleep(0) in C. It could be done in the following way:

in C:

fiber_wakeup(fiber_self());
fiber_yield();

in Lua:

fiber.self():wakeup()
fiber.yield()

To get the same effect in C, one can use fiber_reschedule(). In Lua, it is now impossible to reschedule the current fiber directly in the same event loop iteration. One can reschedule self through a second fiber, but it is strongly discouraged:

local self = fiber.self()
fiber.new(function() self:wakeup() end)
fiber.sleep(0)

Fixed memory leak on each box.on_commit() and box.on_rollback() (gh-6025).

Fixed the lack of testing for non-joinable fibers in fiber_join() call. This could lead to unpredictable results. Note the issue affects C level only, in Lua interface fiber:join() the protection is turned on already.

Now Tarantool yields when scanning .xlog files for the latest applied vclock and when finding the right place in .xlogs to start recovering. This means that the instance is responsive right after box.cfg call even when an empty .xlog was not created on the previous exit. Also, this prevents the relay from timing out when a freshly subscribed replica needs rows from the end of a relatively long (hundreds of MBs) .xlog (gh-5979).

The counter in x.yM rows processed log messages does not reset on each new recovered xlog anymore.

Fixed wrong type specification when printing fiber state change which led to negative fiber’s ID logging (gh-5846).

For example,

main/-244760339/cartridge.failover.task I> Instance state changed

instead of proper

main/4050206957/cartridge.failover.task I> Instance state changed

Fiber IDs were switched to monotonically increasing unsigned 8-byte integers so that there wouldn’t be IDs wrapping anymore. This allows to detect fiber’s precedence by their IDs if needed (gh-5846).

Fixed a crash in JSON update on tuple/space when it had more than one operation, they accessed fields in reversed order, and these fields did not exist. Example: box.tuple.new({1}):update({{'=', 4, 4}, {'=', 3, 3}}) (gh-6069).

Fixed invalid results produced by the json module’s encode function when it was used from Lua’s garbage collector. For instance, in functions used as ffi.gc() (gh-6050).

Added check for user input of the number of iproto threads—value must be > 0 and less than or equal to 1000 (gh-6005).

Fixed error related to the fact that if a user changed the listen address, all iproto threads closed the same socket multiple times.

Fixed error related to Tarantool not deleting the unix socket path when finishing work.

Fixed a crash in MVCC during simultaneous update of a key in different transactions (gh-6131).

Fixed a bug when memtx MVCC crashed during reading uncommitted DDL (gh-5515).

Fixed a bug when memtx MVCC crashed if an index was created in the transaction (gh-6137).

Fixed segmentation fault with MVCC when an entire space was updated concurrently (gh-5892).

Fixed a bug with failed assertion after stress update of the same key (gh-6193).

Fixed a crash that happened when a user called box.snapshot during an incomplete transaction (gh-6229).

Fixed console client connection breakage if request times out (gh-6249).

Added missing broadcast to net.box.future:discard() so that now fibers waiting for a request result are woken up when the request is discarded (gh-6250).

box.info.uuid, box.info.cluster.uuid, and tostring(decimal) with any decimal number in Lua sometimes could return garbage if __gc handlers were used in the user’s code (gh-6259).

Fixed the error message that happened in a very specific case during MVCC operation (gh-6247).

Fixed a repeatable read violation after delete (gh-6206).

Fixed a bug when hash select{} was not tracked by MVCC engine (gh-6040).

Fixed a crash in MVCC after the drop of a space with several indexes (gh-6274).

Fixed a bug when GC at some state could leave tuples in secondary indexes (gh-6234).

Disallowed yields after DDL operations in MVCC mode. It fixes a crash which takes place in case several transactions refer to system spaces (gh-5998).

Fixed a bug in MVCC connected which happened on a rollback after DDL operation (gh-5998).

Fixed a bug when rollback resulted in unserializable behaviour (gh-6325).

At the moment, when a net.box connection is closed, all requests that have not been sent will be discarded. This patch fixes this behavior: all requests queued for sending before the connection is closed are guaranteed to be sent (gh-6338).

Fixed a crash during replace of malformed tuple into _schema system space (gh-6332).

Fixed dropping incoming messages when the connection is closed or SHUT_RDWR received and net_msg_max or readahead limit is reached (gh-6292).

Fixed memory leak in case of replace during background alter of the primary index (gh-6290).

Fixed a bug when rolled back changes appear in the built-in-background index (gh-5958).

Fixed a crash while encoding an error object in the MsgPack format (gh-6431).

Fixed a bug when an index was inconsistent after background build in case the primary index was hash (gh-5977).

Now inserting a tuple with the wrong id` field into the _priv space returns the correct error (gh-6295).

Fixed dirty read in MVCC after space alter (gh-6263, gh-6318).

Fixed a crash in case the fiber changing box.cfg.listen is woken up (gh-6480).

Fixed box.cfg.listen not reverted to the old address in case the new one is invalid (gh-6092).

Fixed a crash caused by a race between box.session.push() and closing connection (gh-6520).

Fixed a bug because of which the garbage collector could remove an xlog file that was still in use (gh-6554).

Fixed crash during granting privileges from guest (gh-5389).

Fixed an error in listening when the user passed uri in numerical form after listening unix socket (gh-6535).

Fixed a crash that could happen in case a tuple is deleted from a functional index while there is an iterator pointing to it (gh-6786).

Fixed memory leak in interactive console (gh-6817).

Fixed an assertion fail when passing a tuple without primary key fields to before_replace trigger. Now tuple format is checked before execution of before_replace triggers and after each one (gh-6780).

Banned DDL operations in space on_replace triggers, since they could lead to a crash (gh-6920).

Implemented constraints and foreign keys. Now users can create function constraints and foreign key relations (gh-6436).

Fixed a bug due to which all fibers created with fiber_attr_setstacksize() leaked until the thread exit. Their stacks also leaked except when fiber_set_joinable(..., true) was used.

Fixed a crash in MVCC related to a secondary index conflict (gh-6452).

Fixed a bug which resulted in wrong space count (gh-6421).

SELECT in RO transaction now reads confirmed data, like a standalone (autocommit) SELECT does (gh-6452).

Fixed a crash when Tarantool was launched with multiple -e or -l options without a space between the option and the value (gh-5747).

Fixed effective session and user not propagated to box.on_commit and box.on_rollback trigger callbacks (gh-7005).

Fixed usage of box.session.peer() in box.session.on_disconnect() trigger. Now, it’s safe to assume that box.session.peer() returns the address of the disconnected peer, not nil, as it used to (gh-7014).

Fixed creation of a space with a foreign key pointing to the same space (gh-6961).

Fixed a bug when MVCC failed to track nothing-found range select (gh-7025).

Allowed complex foreign keys with null fields (gh-7046).

Added decoding of election messages: RAFT and PROMOTE to xlog Lua module (gh-6088). Otherwise tarantoolctl shows plain number in type

HEADER:
  lsn: 1
  replica_id: 4
  type: 31
  timestamp: 1621541912.4592

instead of symbolic representation

HEADER:
  lsn: 1
  replica_id: 4
  type: PROMOTE
  timestamp: 1621541912.4592

[Breaking change] Return value signedness of 64-bit time functions in clock and fiber was changed from uint64_t to int64_t both in Lua and C (gh-5989).

Intervals received after datetime arithmetic operations may be improperly normalized if the result was negative

tarantool> date.now() - date.now()
---
- -1.000026000 seconds
...

i.e. 2 immediately called date.now() produce very close values, which difference should be close to 0, not 1 second (gh-6882).