Date: 2023-09-12
Tag: 3.0.0-alpha3

## Overview

3.0.0-alpha3 is the [beta][release_policy] version of the 3.0 release series.

This release introduces 71 new features and resolves 23 bugs since
the 2.11.1 version. There can be bugs in less common areas. If you find any,
feel free to [report an issue][issues] on GitHub.

Notable changes are:

* Added initial support of configuring vshard's storage and router roles in the
  declarative configuration (gh-9007).
* Fixes and improvements of the configuration process based on early adopters feedback.
* New config sections: metrics, feedback, vinyl, flightrec (EE), security (EE) (gh-8861).
* Added initial support of roles in the declarative configuration.

[release_policy]: https://www.tarantool.io/en/doc/latest/dev_guide/release_management/#release-policy
[issues]: https://github.com/tarantool/tarantool/issues

## Compatibility

Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data
layout, client-server protocol, and replication protocol.

Please [upgrade][upgrade] using the `box.schema.upgrade()` procedure to unlock
all the new features of the 2.x series.

[upgrade]: https://www.tarantool.io/en/doc/latest/book/admin/upgrades/

## Functionality added or changed

### Core

- **[Breaking change]** Disabled DDL operations with an old system schema.
  Now you have to run `box.schema.upgrade()` before you can execute any DDL
  operations (gh-7149).
* Now `index_object.parts` contains the following methods, similar to the
  `key_def` Lua module: `extract_key()`, `compare()`, `compare_with_key()`,
  `merge()` (gh-7356).
* Introduced the `box.watch_once()` function to get the value currently
  associated with a notification key on the local instance without subscribing
  to future changes (gh-6493).
* Introduced the new `IPROTO_WATCH_ONCE` request to get the value currently
  associated with a notification key on a remote instance without subscribing
  to future changes (gh-6493).
* Introduced the `conn:watch_once()` net.box connection method to get the value
  currently associated with a notification key on a remote instance without
  subscribing to future changes. The new method is implemented using the
  `IPROTO_WATCH_ONCE` request type (gh-6493).
* Introduced the default field values in the space format (gh-8157).
* Added support for functional variant of the default field values (gh-8609).

### Memtx

* Added memtx read view statistics to `box.stat.memtx()` (gh-8501).

### Vinyl

* Introduced the `memory.tuple` statistic for `box.stat.vinyl()` that shows
  the total size of memory occupied by all tuples allocated by the Vinyl engine
  (gh-8485).

### Replication

* Added a new `bootstrap_leader` configuration option to specify the node from
  which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy`
  to `'config'` and set `bootstrap_leader` value to either the URI or UUID of
  the desired bootstrap leader. For example:
  ```lua
    box.cfg{
        bootstrap_strategy = 'config',
        bootstrap_leader = 'localhost:3301',
        replication = {
            'localhost:3301',
            'localhost:3302',
        },
        listen = '3302',
    }
  ```
  (gh-7999).
* You may now control which node new replicas choose as a bootstrap leader
  without touching node config. To do so, set `box.cfg.bootstrap_strategy` to
  `'supervised'`, and the nodes will only bootstrap off the node on which you
  called `box.ctl.make_bootstrap_leader()` last.
  This works on an empty replica set bootstrap as well: start the admin console
  before configuring the nodes. Then configure the nodes:
  ```lua
  box.cfg{
      bootstrap_strategy = 'supervised',
      replication = ...,
      listen = ...,
  }
  ```
  Finally, call `box.ctl.make_bootstrap_leader()` through the admin console
  on the node you want to promote. All the nodes will bootstrap off that node
  (gh-8509).
* A new option `box.cfg.cluster_name` allows assigning a human-readable name to
  the entire cluster. It has to match in all instances and is displayed in
  `box.info.cluster.name` (gh-5029).

* A new option `box.cfg.replicaset_name` allows assigning a human-readable name
  to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value
  must be the same across all instances of one replicaset. The replicaset name
  is displayed in `box.info.replicaset.name` (gh-5029).

* A new option `box.cfg.instance_name` allows assigning a human-readable name to
  the instance. It works the same as `box.cfg.instance_uuid`. Its value must be
  unique in the replicaset. The instance name is displayed in `box.info.name`.
  Names of other replicas in the same replicaset are visible in
  `box.info.replication[id].name` (gh-5029).

* Instance at rebootstrap can change its UUID while keeping its numeric ID if it
  has the same non-empty instance name (gh-5029).
* Added the ability to set the `bootstrap_leader` configuration option to the
  instance name of the desired bootstrap leader:
  ```lua
    box.cfg{
        bootstrap_strategy = 'config',
        bootstrap_leader = 'leader-name',
        replication = {
            ...
        },
        ...
    }
  ```
  (gh-7999, gh-8539).

### LuaJIT

* LuaJIT now can handle exceptions on traces (gh-7745).

### Lua

* **[Breaking change]** Added the new `varbinary` type to Lua. An object of
  this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so
  it can be used for storing binary blobs in the database. This also works the
  other way round: data fields stored as `MP_BIN` are now decoded in Lua as
  varbinary objects, not as plain strings, as they used to be. Since the latter
  may cause compatibility issues, the new compat option `binary_data_decoding`
  was introduced to revert the built-in decoder to the old behavior (gh-1629).
* Introduced the new Lua module `trigger`. This module enables managing and
  calling triggers stored in trigger registry (gh-8656).

#### Netbox

* Introduced the new `net.box` module function `from_fd` for creating a new
  connection from a socket file descriptor number (gh-8984).

#### Popen

* Introduced new option `inherit_fds` for `popen.new`. The option takes
  an array of file descriptor numbers that should be left open in the child
  process (gh-8926).

#### Socket

* Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and
  `socket:detach` (gh-8927).

### SQL
* Introduced the `SHOW CREATE TABLE` statement (gh-8098).

### Box

* Non-system spaces in a snapshot file are now sorted by their IDs.
  As before, all non-system spaces are stored after system spaces (gh-7954).
* **[Breaking change]** Dropped support for `IPROTO_CALL16` from `net.box`
  (i.e., the `call_16` option and the `call_16` method for `net.box`
  connections) (gh-8636).
* Added support for square brackets in procedure resolution for Lua calls
  (gh-8604).
* Added support for accepting IPROTO requests with a space or index name instead
  of an identifier (gh-8146).
* Added a new `box.info` parameter `hostname` (gh-8605).
* Added the `box.cfg.memtx_sort_threads` parameter that specifies the number of
  threads used to sort indexes keys on loading a memtx database. OpenMP is
  not used to sort keys anymore (gh-3389).
* Changed the output of the `fselect` index method from a table of strings to
  a multi-line string and dropped the `print` and `use_nbsp` options. With the
  `yaml_pretty_multiline` compat option enabled by default, multi-line strings
  now look good in the console, so there's no need to return a table of strings
  to prettify the `fselect` output anymore.
* Allowed multiple updates to the same tuple field in a single call (gh-8226).
* Introduced `box.tuple.format` that enables format definition for tuples
  created via `box.tuple.new` (standalone tuples) (gh-4693).
* **[Breaking change]** Disabled argument list syntax of `box.tuple.new` (this
  was needed for gh-4693). It is possible to switch to the old behavior using
  the compatibility option `box_tuple_new_vararg`.
* Sped up `index.select` and `index.pairs` with the `after` option by up to 30%
  in a synthetic test by eliminating an extra buffer allocation.
* The function `box.session.push` is now deprecated. Consider using
  `box.broadcast` instead (gh-8802).
* Introduced the new function `box.session.new` for creating a new IPROTO
  session from a socket file descriptor number (gh-8801).
* Introduced the new `lua_eval` and `lua_call` object types for
  `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval`
  allows the user to execute an arbitrary Lua expression with the
  `IPROTO_EVAL` request. Granting the `'execute'` privilege on `lua_call`
  allows the user to execute any global user-defined Lua function with
  the `IPROTO_CALL` request (gh-8803).
* **[Breaking change]** Introduced the new `sql` object type for
  `box.schema.user.grant`. Now only users with the `'execute'` privilege
  granted on `sql` or `universe` can execute SQL expressions with the
  `IPROTO_EXECUTE` or `IPROTO_PREPARE` requests. To revert to the old behavior
  (no SQL access checks), use the `sql_priv` compat option (gh-8803).
* The maximum length of `box.cfg{}` string parameters is now 512 instead of 256.
* Added a new flag `tarantool.build.asan` that shows whether build
  flag `ENABLE_ASAN` is set.
* Added the `leader_name` field to `box.info.election` (gh-8931).

### Cli

* **[Behavior change]** Disabled starting the Lua REPL by default when running
  Tarantool. Now, Tarantool yields the message that shows the command usage.
  To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via
  `stdin`, use dash (`-`) as the script name. For more information see a help
  message by running `tarantool -h` (gh-8613).
* Introduced new command-line options `--name` and `--config` (gh-8613).

### Compat

* The following compatibility options' defaults were switched to new behavior:
    * `yaml_pretty_multiline`
    * `sql_seq_scan_default`
    * `json_escape_forward_slash`
    * `fiber_channel_close_mode`
    * `fiber_slice_default`
    * `box_cfg_replication_sync_timeout`

  More information on the new behavior can be found on the [Module compat](https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/) page.

### Config

* Initial version of the declarative server and cluster configuration
  (gh-8724).
* All vinyl options are now supported (gh-8861).
* All feedback options are now supported (gh-8861).
* The `memtx.sort_threads` option is now supported (gh-8861).
* The `bootstrap_leader` option is now supported (gh-8861).
* All flight recorder options are now supported (gh-8861).
* All security hardening `box.cfg{}` options are now supported by
  config (gh-8861).
* Added a low priority environment configuration source, which looks into
  `TT_*_DEFAULT` variables. It is useful to declare defaults (gh-8862).
* It is now possible to access configuration from the application script using
  the `config:get()` method (gh-8862).
* All metrics configuration options from `box.cfg{}` are now supported
  in the YAML config (gh-8861).
* Added `--help-env-list` CLI option (gh-8862).
* Introduced a non-public API for extending the declarative config
  functionality in Tarantool Community Edition (gh-8862).
* Improved the credentials applier: now it supports two-way synchronization
  of roles and privileges for both users and roles (gh-8861).
* Removed the sha1 and sha256 hash entries from the config credentials
  schema as not planned for development (gh-8967).
* Implemented a full password support in the `config.credentials` schema,
  including a password setting, updating and removal for the `chap-sha1`
  auth type (supported by both Tarantool Community Edition and Tarantool
  Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition
  where it is available) (gh-8967).
* A group name from a topology defined by the cluster configuration doesn't
  persisted anymore as `box.cfg.cluster_name` (gh-8862).
* The default directory and file paths are changed to `var/run/<...>`,
  `var/log/<...>`, `var/lib/<...>` and so on (gh-8862).
* The directory that contains the configuration file is now added into the Lua
  modules search paths (gh-8862).
* Most of vshard options are now added in the config (gh-9007).
* Added initial support of vshard (gh-9007).
* All audit options are now supported (gh-8861).
* Non-default privileges are now revoked from default users and roles
  when they are removed from the config (gh-8967).
* Introduced the initial support for roles - programs that run when
  a configuration is loaded or reloaded (gh-9078).

### Console

* Now multiline commands can be used by setting continuation marker
  using the `\set continuation` command.

### Debugger

* Introduced readline support to tarantool debugger (gh-7738).

### Space

* Introduced space type: a new space definition field.
* Introduced the fully temporary space type. It is the same as data-temporary
  but also has temporary metadata. Temporary spaces can now be created in
  read_only mode, they disappear after server restart and don't exist on
  replicas (gh-8323).

### Test

#### Fuzz

* Added a grammar-based SQL fuzzer (gh-4826).

### Build

* Tarantool does not depend on libgomp anymore (gh-7689).
* Added building static `deb` and `rpm` packages.
* Dropped building distro-specific `deb` and `rpm` Tarantool packages.
* Dropped building Tarantool packages with GC32. Now only GC64 packages
  are shipped.
* Exported all LuaJIT public functions from Tarantool binary (gh-3680).

## Bugs fixed

### Core

* **[Breaking change]** The key `box.space._schema['cluster']` is renamed to
  `'replicaset_uuid'`. That is not expected to be breaking because `_schema` is
  an internal system space, but the key was visible in public and documented
  (gh-5029).
* **[Breaking change]** The table `box.info.cluster` is renamed to
  `box.info.replicaset`. The behaviour can be reverted using the `compat` option
  [box_info_cluster_meaning](https://tarantool.io/compat/box_info_cluster_meaning)
  (gh-5029).
* Eliminated implicit conversion of unprintable utf-8 strings to binary blobs
  when encoded in YAML. Now unprintable characters are encoded as escaped utf-8
  code points, for example, `\x80` or `\u200B` (gh-8756).
* Fixed decoding of escape sequences for single-byte character codes from YAML.
  Before the fix, single-byte character codes between `0x80` and `0xff` would
  be erroneously converted to two-byte UTF-8 code points, for example, `\x80`
  would be decoded as `\uC280` (gh-8782).
* Now `index_object.parts.exclude_null` always contains `false` rather than
  `null` when it is actually `false` (gh-8649).
* Fixed a crash that could happen when inserting an index definition with an
  empty parts list directly into `box.space._index` (gh-8688).
* Fixed the memory leak on unpacking an invalid MsgPack error extension
  (gh-8921).
* Fixed a use-after-free bug in fiber recycling code (gh-9020).
* Fixed a use-after-free bug in iproto server code (gh-9037).
* Fixed a heap-buffer-overflow bug in fiber creation code (gh-9026).
* An error is no longer raised on an attempt to print a dead fiber (gh-4265).
* Fixed a possible out-of-bound write in debug build on unpacking a MsgPack
  error extension (gh-9098).

### Replication

* Fixed a bug that allowed to change replicaset UUID via `_schema` space.
* Fixed a bug when new instances could try to register via an anon instance
  which previously failed to apply `box.cfg{replication_anon = false}`.
* Fixed a bug when `box.ctl.demote()` with `box.cfg{election_mode = 'off'}`
  and an owned synchro queue could simply not do anything (gh-6860).
### LuaJIT

Backported patches from the vanilla LuaJIT trunk (gh-8825).
The following issues were fixed as part of this activity:

* Fixed `BC_UCLO` insertion for returns.
* Fixed recording of `BC_VARG` with unused vararg values.
* Initialization instructions on trace are now emitted only for the first
  member of a union.
* Prevent integer overflow while parsing long strings.
* Fixed various `^` operator and `math.pow()` function inconsistencies.
* Fixed parsing with predicting `next()` and `pairs()`.
* Fixed binary number literal parsing. Parsing of binary number with a zero
  fractional part raises error too now.
* Fixed load forwarding optimization applied after table rehashing.
* Fixed recording of the `BC_TSETM`.

### Lua

#### Fio

* The default permission mode for `fio.open()` was changed for newly
  created files to 0666 (before umask) (gh-7981).

### SQL

* Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause
  (ghs-119).

### Box

- Fixed the ability to drop `_vinyl_deferred_delete` system space (gh-5279).
* Fixed a bug when it was not possible to access spaces named `_stream` and
  `_stream_space_cache` and indexes named `_space` using the `net.box` stream
  objects (gh-8598).
* **[Breaking change]** The `cord_slab_cache` symbol was removed from the public
  API export (gh-7124).

### Config

* Fixed an error message if the cluster configuration was not provided or the
  instance was not found in the cluster configuration during reload (gh-8862).
* Support parent directories creation for options that accept a directory or a
  file (gh-8862).
* Create parent directories for `console.socket` and `log.file` (gh-8862).
* Create the `process.work_dir` directory (gh-8862).
* Consider all the paths as relative to `process.work_dir` when creating
  necessary directories (gh-8862).
