Date: 2021-08-20

Tag: 2.10.0-beta1

## Overview

2.10.0-beta1 is the [beta][release_policy] version of the 2 release series.

This release introduces 28 new features and resolves 56 bugs since
the 2.8 version. There can be bugs in less common areas. If you find any,
feel free to [report an issue][issues] on GitHub.

Notable changes are:

* Preliminary support of ARM64 architecture (GNU/Linux).
* Preliminary support of Apple M1 architecture (macOS).
* Consistent SQL type system.
* Lightning fast net.box module.
* Interative transactions in binary protocol.
* Small tuples are now stored more efficiently.

[release_policy]: https://www.tarantool.io/en/doc/latest/release/policy/
[issues]: https://github.com/tarantool/tarantool/issues

## Compatibility

Tarantool 2.x is backward compatible with Tarantool 1.10.x in the binary data
layout, client-server protocol, and replication protocol.

[Upgrade][upgrade] using the `box.schema.upgrade()` procedure to unlock
all the new features of the 2.x series.

[upgrade]: https://www.tarantool.io/en/doc/latest/book/admin/upgrades/

## Functionality added or changed

### Core

* Field type UUID is now part of field type SCALAR (gh-6042).

* Field type UUID is now available in SQL, and new UUID can be generated using new SQL built-in function uuid() (gh-5886).

* **[Breaking change]** timeout() method of net.box connection, which was
  marked deprecated more than four years ago (in 1.7.4) was dropped, because
  it negatively affected performance of hot net.box methods, like call() and
  select(), in case those are called without specifying a timeout (gh-6242).

* **[Breaking change]**: timeout() method of net.box connection was dropped.

* Improved net.box performance by up to 70% by rewriting hot code paths in C
(gh-6241).

* Introduce compact tuples that allow to save 4 bytes per tuple in case of small userdata  (gh-5385)

* Streams and interactive transactions over streams are implemented
  in iproto. Stream is associated with it's ID, which is unique within
  one connection. All requests with same not zero stream ID belongs to
  the same stream. All requests in stream processed synchronously. The
  execution of the next request will not start until the previous one is
  completed. If request has zero stream ID it does not belong to stream
  and is processed in the old way.
  In `net.box`, stream is an object above connection that has the same
  methods, but allows to execute requests sequentially. ID is generated
  on the client side automatically. If user writes his own connector and
  wants to use streams, he must transmit stream_id over iproto protocol.
  The main purpose of streams is transactions via iproto. Each stream
  can start its own transaction, so they allows multiplexing several
  transactions over one connection. There are multiple ways to begin,
  commit and rollback transaction: using appropriate stream methods, using
  `call` or `eval` methods or using `execute` method with sql transaction
  syntax. User can mix these methods, for example, start transaction using
  `stream:begin()`, and commit transaction using `stream:call('box.commit')`
  or stream:execute('COMMIT').
  If any request fails during the transaction, it will not affect the other
  requests in the transaction. If disconnect occurs when there is some active
  transaction in stream, this transaction will be rollbacked, if it does not
  have time to commit before this moment.

* Add new 'memtx_allocator' option to box.cfg{} which allows to
  select the appropriate allocator for memtx tuples if necessary.
  Possible values are "system" for malloc allocator and "small"
  for default small allocator.
  Implement system allocator, based on malloc: slab allocator, which
  is used for tuples allocation, has a certain disadvantage - it tends
  to unresolvable fragmentation on certain workloads (size migration).
  In this case user should be able to choose other allocator. System
  allocator based on malloc function, but restricted by the same qouta
  as slab allocator. System allocator does not alloc all memory at start,
  istead, it allocate memory as needed, checking that quota is not exceeded
  (gh-5419).

### Replication

* Introduced `box.info.replication[n].downstream.lag` field to monitor
   state of replication. This member represents a lag between the main
   node writes a certain transaction to it's own WAL and a moment it
   receives an ack for this transaction from a replica (gh-5447).

* Introduce on_election triggers. The triggers may be registered via
  `box.ctl.on_election()` interface and are run asynchronously each time
  `box.info.election` changes (gh-5819).

### LuaJIT

* Introduced support for `LJ_DUALNUM` mode in luajit-gdb.py (gh-6224).

* Introduced preliminary support of GNU/Linux ARM64 and MacOS M1. In scope of
  this activity the following issues have been resolved:
  - Introduced support for full 64-bit range of lightuserdata values (gh-2712)
  - Fixed memory remapping issue when the page leaves 47-bit segments
  - Fixed M1 architecture detection (gh-6065)
  - Fixed variadic arguments handling in FFI on M1 (gh-6066)
  - Fixed `table.move` misbehaviour when table reallocation occurs (gh-6084)
  - Fixed Lua stack inconsistency when xpcall is called with invalid second
    argument on ARM64 (gh-6093)
  - Fixed `BC_USETS` bytecode semantics for closed upvalues and gray strings
  - Fixed side exit jump target patching considering the range values of the
    particular instruction (gh-6098)
  - Fixed current Lua coroutine restoring on exceptional path on ARM64 (gh-6189)

### Lua

* Introduce method `table.equals`. It compares 2 tables by value and respects
  `__eq` metamethod.

#### Digest

* Introduce new hash types in digest module - `xxhash32` and `xxhash64`
   (gh-2003).

#### Fiber

* Introduce `fiber_object:info()` to get `info` from fiber. Works as `require(fiber).info()` but only for one fiber.

* Introduce `fiber_object:csw()` to get `csw` from fiber (gh-5799).

* Changed `fiber.info()` to hide backtraces of idle fibers (gh-4235).

#### Log

* Implemented support of symbolic log levels representation
   in `log` module (gh-5882). Now it is possible to specify
   levels the same way as in `box.cfg{}` call. For example
   instead of
   ``` Lua
   require('log').cfg{level = 6}
   ```
   One can use
   ``` Lua
   require('log').cfg{level = 'verbose'}
   ```

### SQL

* Descriptions of type mismatch error and inconsistent type error become more
  informative (gh-6176).

* Removed explicit cast from BOOLEAN to numeric types and vice versa (gh-4770).

* Removed explicit cast from VARBINARY to numeric types and vice versa (gh-4772,
  gh-5852).

* Fixed a bug due to which a string that is not NULL terminated could not be
  cast to BOOLEAN, even if the conversion should be successful according to the
  rules.

* Now a numeric value can be cast to another numeric type only if the cast is
  precise. In addition, a UUID value cannot be implicitly cast to
  STRING/VARBINARY, and a STRING/VARBINARY value cannot be implicitly cast to
  a UUID (gh-4470).

* Now any number can be compared to any other number, and values of any scalar
  type can be compared to any other value of the same type. A value of a
  non-numeric scalar type cannot be compared with a value of any other scalar
  type (gh-4230).

* SQL built-in functions were removed from \_func system space (gh-6106).

* Function are now looked up first in SQL built-in functions and then in
  user-defined functions.

* Fixed incorrect error message in case of misuse of the function used to set
  the default value.

* The typeof() function with NULL as an argument now returns "NULL" (gh-5956).

* The SCALAR and NUMBER types have been reworked in SQL. Now SCALAR values
  cannot be implicitly cast to any other scalar type, and NUMBER values cannot be
  implicitly cast to any other numeric type. This means that arithmetic and
  bitwise operations and concatenation are no longer allowed for SCALAR and NUMBER
  values. In addition, any SCALAR value can now be compared with values of any
  other scalar type using the SCALAR rules (gh-6221).

* Field type DECIMAL is now available in SQL. Decimal can be implcitly cast to
  and from INTEGER and DOUBLE, it can participate in arithmetic operations and
  comparison between DECIMAL and all other numeric types are defined (gh-4415).

* The argument types of SQL built-in functions are now checked in most cases
  during parsing. In addition, the number of arguments is now always checked
  during parsing (gh-6105).

### Fiber

* Previously csw (Context SWitch) of new fiber can be greater than 0, now it is always 0 (gh-5799).

### Luarocks

* Set FORCE_CONFIG=false for luarocks config to allow loading project-side .rocks/config-5.1.lua

### Build

* Fedora 34 builds are now supported. (gh-6074)

* Fedora 28 and 29 builds are no longer supported.

## Bugs fixed

### Core

* **[Breaking change]** `fiber.wakeup()` in Lua and `fiber_wakeup()` in C became
  NOP on the currently running fiber. Previously they allowed to "ignore" the
  next yield or sleep leading to unexpected spurious wakeups. Could lead to a
  crash (in debug build) or undefined behaviour (in release build) if called
  right before `fiber.create()` in Lua or `fiber_start()` in C (gh-6043).

  There was a single usecase for that - reschedule in the same event loop
  iteration which is not the same as `fiber.sleep(0)` in Lua and
  `fiber_sleep(0)` in C. Could be done in C like that:
  ```C
  fiber_wakeup(fiber_self());
  fiber_yield();
  ```
  and in Lua like that:
  ```Lua
  fiber.self():wakeup()
  fiber.yield()
  ```
  Now to get the same effect in C use `fiber_reschedule()`. In Lua it is now
  simply impossible to reschedule the current fiber in the same event loop
  iteration directly. But still can reschedule self through a second fiber like
  this (**never use it, please**):
  ```Lua
  local self = fiber.self()
  fiber.new(function() self:wakeup() end)
  fiber.sleep(0)
  ```

* Fixed memory leak on each `box.on_commit()` and `box.on_rollback()` (gh-6025).

* Fixed lack of testing for non noinable fibers in `fiber_join()` call.
  This could lead to unpredictable results. Note the issue affects C
  level only, in Lua interface `fiber:join()` the protection is
  turned on already.

* Now tarantool yields when scanning `.xlog` files for the latest applied vclock
  and when finding the right place in `.xlog`s to start recovering. This means
  that the instance is responsive right after `box.cfg` call even when an empty
  `.xlog` was not created on previous exit.
  Also this prevents relay from timing out when a freshly subscribed replica
  needs rows from the end of a relatively long (hundreds of MBs) `.xlog`
  (gh-5979).

* The counter in `x.yM rows processed` log messages does not reset on each new
  recovered `xlog` anymore.

* Fixed wrong type specification when printing fiber state
   change which lead to negative fiber's ID logging (gh-5846).

   For example
   ```
   main/-244760339/cartridge.failover.task I> Instance state changed
   ```
   instead of proper
   ```
   main/4050206957/cartridge.failover.task I> Instance state changed
   ```

* Fiber IDs are switched to monotonically increasing unsigned 8 byte
  integers so that there won't be IDs wrapping anymore. This allows
  to detect fiber's precedence by their IDs if needed (gh-5846).

* Fixed a crash in JSON update on tuple/space when it had more than one
  operation, they accessed fields in reversed order, and these fields didn't
  exist. Example: `box.tuple.new({1}):update({{'=', 4, 4}, {'=', 3, 3}})`
  (gh-6069).

* Fixed invalid results produced by `json` module's `encode` function when it
  was used from Lua's garbage collector. For instance, in functions used as
  `ffi.gc()` (gh-6050).

* Added check for user input of the number of iproto threads - value
   must be > 0 and less then or equal to 1000 (gh-6005).

* Fixed error, related to the fact, that if user changed listen address,
  all iproto threads closed same socket multiple times.
  Fixed error, related to the fact, that tarantool not deleting the unix
  socket path, when it's finishing work.

* Fixed a crash in MVCC during simultaneous update of a key in different transactions (gh-6131)

* Fixed a bug when memtx mvcc crashed during reading uncommitted DDL (gh-5515)

* Fixed a bug when memtx mvcc crashed if an index was created in transaction (gh-6137)

* Fixed segmentation fault with mvcc when entire space was updated concurrently (gh-5892)

* Fixed a bug with failed assertion after stress update of the same key. (gh-6193)

* Fixed a crash if you call box.snapshot during an incomplete transaction (gh-6229)

* Fixed console client connection breakage if request times out (gh-6249).

* Added missing broadcast to net.box.future:discard() so that now fibers
  waiting for a request result are woken up when the request is discarded
  (gh-6250).

* `box.info.uuid`, `box.info.cluster.uuid`, and `tostring(decimal)` with any
   decimal number in Lua sometimes could return garbage if `__gc` handlers are
   used in user's code (gh-6259).

* Fix an error message that happened in very specific case
  during mvcc operation (gh-6247)

* Fixed a repeatable read violation after delete (gh-6206)

* Fixed a bug when hash select{} was not tracked by mvcc engine (gh-6040)

* Fixed a crash in mvcc after drop of a space with several indexes (gh-6274)

* Fixed a bug when GC at some state could leave tuples in secondary indexes  (gh-6234)

* Disallow yields after DDL operations in MVCC mode. It fixes crash which takes
  place in case several transactions refer to system spaces (gh-5998).

* Fixed bug in MVCC connected which happens on rollback after
  DDL operation (gh-5998).

* Fix a bug when rollback resulted in unserializable behaviour (gh-6325)

### Vinyl

* Fixed possible keys divergence during secondary index build which might
  lead to missing tuples in it (gh-6045).

* Fixed a race between Vinyl garbage collection and compaction resulting in
  broken vylog and recovery (gh-5436).

### Replication

* Fixed use after free in relay thread when using elections (gh-6031).

* Fixed a possible crash when a synchronous transaction was followed by an
  asynchronous transaction right when its confirmation was being written
  (gh-6057).

* Fixed an error when a replica, at attempt to subscribe to a foreign cluster
  (with different replicaset UUID), didn't notice it is not possible, and
  instead was stuck in an infinite retry loop printing an error about "too
  early subscribe" (gh-6094).

* Fixed an error when a replica, at attempt to join a cluster with exclusively
  read-only replicas available, instead of failing or retrying just decided to
  boot its own replicaset. Now it fails with an error about the other nodes
  being read-only so they can't register it (gh-5613).

* When an error happened during appliance of a transaction received from a
  remote instance via replication, it was always reported as "Failed to write
  to disk" regardless of what really happened. Now the correct error is shown.
  For example, "Out of memory", or "Transaction has been aborted by conflict",
  and so on (gh-6027).

* Fixed replication stopping occasionally with `ER_INVALID_MSGPACK` when replica
  is under high load (gh-4040).

* Fixed a cluster sometimes being unable to bootstrap if it contains nodes with
  `election_mode` `manual` or `voter` (gh-6018).

* Fixed a possible crash when `box.ctl.promote()` was called in a cluster with
  >= 3 instances, happened in debug build. In release build it could lead to
  undefined behaviour. It was likely to happen if a new node was added shortly
  before the promotion (gh-5430).

* Fixed a rare error appearing when MVCC (`box.cfg.memtx_use_mvcc_engine`) was
  enabled and more than one replica was joined to a cluster. The join could fail
  with the error `"ER_TUPLE_FOUND: Duplicate key exists in unique index
  'primary' in space '_cluster'"`. The same could happen at bootstrap of a
  cluster having >= 3 nodes (gh-5601).

### Raft

* Fixed a rare crash with the leader election enabled (any mode except `off`),
  which could happen if a leader resigned from its role at the same time as some
  other node was writing something related to the elections to WAL. The crash
  was in debug build and in the release build it would lead to undefined
  behaviour (gh-6129).

* Fixed an error when a new replica in a Raft cluster could try to join from a
  follower instead of a leader and failed with an error `ER_READONLY` (gh-6127).

### LuaJIT

* Fixed optimization for single-char strings in `IR_BUFPUT` assembly routine.

* Fixed slots alignment in `lj-stack` command output when `LJ_GC64` is enabled
  (gh-5876).

* Fixed dummy frame unwinding in `lj-stack` command.

* Fixed detection of inconsistent renames even in the presence of sunk values
  (gh-4252, gh-5049, gh-5118).

* Fixed the order VM registers are allocated by LuaJIT frontend in case of
  `BC_ISGE` and `BC_ISGT` (gh-6227).

### Lua

* Fixed a bug when multibyte characters broke `space:fselect()` output.

* When error is raised during encoding call results, auxiliary lightuserdata
  value is not removed from the main Lua coroutine stack. Prior to the fix it
  leads to undefined behaviour during the next usage of this Lua coroutine
  (gh-4617).

* Fixed Lua C API misuse, when the error is raised during call results encoding
  on unprotected coroutine and expected to be catched on the different one, that
  is protected (gh-6248).

#### Triggers

* Fixed possibility crash in case when trigger removes itself.
  Fixed possibility crash in case when someone destroy trigger,
  when it's yield (gh-6266).

### SQL

* User-defined functions can now return VARBINARY to SQL as result (gh-6024).

* Fixed assert on cast of DOUBLE value that greater than -1.0 and less than 0.0
  to INTEGER and UNSIGNED (gh-6255).

* Removed spontaneous conversion from INTEGER to DOUBLE in a field of type
  NUMBER (gh-5335).

* All arithmetic operations can now only accept numeric values (gh-5756).

* Now function quote() will return the argument in case the argument is DOUBLE.
  The same for all other numeric types. For types other than numeric, STRING
  will be returned (gh-6239).

* The TRIM() function now does not lose collation when executed with the
  keywords BOTH, LEADING, or TRAILING (gh-6299).

### MVCC

* Fixed MVCC interaction with ephemeral spaces: TX manager now ignores such
  spaces (gh-6095).

* Fixed a loss of tuple after a conflict exception (gh-6132)

* Fixed a segfault in update/delete of the same tuple (gh-6021)
