Date: 2023-11-20
Tag: 3.0.0-beta1

## Overview

3.0.0-beta1 is the [beta][release_policy] version of the 3.0 release series.

This release introduces 95 new features and resolves 51 bugs since
the 2.11 version. There can be bugs in less common areas. If you find any,
feel free to [report an issue][issues] on GitHub.

Notable changes are:

* New configuration capabilities (gh-8724)
* Fixes and improvements of the configuration process based on early adopters feedback.

[release_policy]: https://www.tarantool.io/en/doc/latest/dev_guide/release_management/#release-policy
[issues]: https://github.com/tarantool/tarantool/issues

## Compatibility

Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data
layout, client-server protocol, and replication protocol.

Please [upgrade][upgrade] using the `box.schema.upgrade()` procedure to unlock
all the new features of the 2.x series.

[upgrade]: https://www.tarantool.io/en/doc/latest/book/admin/upgrades/

## Functionality added or changed

### Core

- **[Breaking change]** Disabled DDL operations with an old system schema.
  Now you have to run `box.schema.upgrade()` before you can execute any DDL
  operations (gh-7149).
* Now `index_object.parts` contains the following methods, similar to the
  `key_def` Lua module: `extract_key()`, `compare()`, `compare_with_key()`,
  `merge()` (gh-7356).
* Introduced the `box.watch_once()` function to get the value currently
  associated with a notification key on the local instance without subscribing
  to future changes (gh-6493).
* Introduced the new `IPROTO_WATCH_ONCE` request to get the value currently
  associated with a notification key on a remote instance without subscribing
  to future changes (gh-6493).
* Introduced the `conn:watch_once()` net.box connection method to get the value
  currently associated with a notification key on a remote instance without
  subscribing to future changes. The new method is implemented using the
  `IPROTO_WATCH_ONCE` request type (gh-6493).
* Introduced the default field values in the space format (gh-8157).
* Added support for functional variant of the default field values (gh-8609).
* `box.schema` can now be accessed before `box.cfg()`.
* Introduced new methods `tuple:info()` and `space:stat()` with the detailed
  information on memory consumed by data (gh-6762).

### Memtx

* Added memtx read view statistics to `box.stat.memtx()` (gh-8501).

### Vinyl

* Introduced the `memory.tuple` statistic for `box.stat.vinyl()` that shows
  the total size of memory occupied by all tuples allocated by the Vinyl engine
  (gh-8485).

### Replication

* Added a new `bootstrap_leader` configuration option to specify the node from
  which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy`
  to `'config'` and set `bootstrap_leader` value to either the URI or UUID of
  the desired bootstrap leader. For example:
  ```lua
    box.cfg{
        bootstrap_strategy = 'config',
        bootstrap_leader = 'localhost:3301',
        replication = {
            'localhost:3301',
            'localhost:3302',
        },
        listen = '3302',
    }
  ```
  (gh-7999).
* You may now control which node new replicas choose as a bootstrap leader
  without touching node config. To do so, set `box.cfg.bootstrap_strategy` to
  `'supervised'`, and the nodes will only bootstrap off the node on which you
  called `box.ctl.make_bootstrap_leader()` last.
  This works on an empty replica set bootstrap as well: start the admin console
  before configuring the nodes. Then configure the nodes:
  ```lua
  box.cfg{
      bootstrap_strategy = 'supervised',
      replication = ...,
      listen = ...,
  }
  ```
  Finally, call `box.ctl.make_bootstrap_leader()` through the admin console
  on the node you want to promote. All the nodes will bootstrap off that node
  (gh-8509).
* A new option `box.cfg.cluster_name` allows assigning a human-readable name to
  the entire cluster. It has to match in all instances and is displayed in
  `box.info.cluster.name` (gh-5029).

* A new option `box.cfg.replicaset_name` allows assigning a human-readable name
  to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value
  must be the same across all instances of one replicaset. The replicaset name
  is displayed in `box.info.replicaset.name` (gh-5029).

* A new option `box.cfg.instance_name` allows assigning a human-readable name to
  the instance. It works the same as `box.cfg.instance_uuid`. Its value must be
  unique in the replicaset. The instance name is displayed in `box.info.name`.
  Names of other replicas in the same replicaset are visible in
  `box.info.replication[id].name` (gh-5029).

* Instance at rebootstrap can change its UUID while keeping its numeric ID if it
  has the same non-empty instance name (gh-5029).
* Added the ability to set the `bootstrap_leader` configuration option to the
  instance name of the desired bootstrap leader:
  ```lua
    box.cfg{
        bootstrap_strategy = 'config',
        bootstrap_leader = 'leader-name',
        replication = {
            ...
        },
        ...
    }
  ```
  (gh-7999, gh-8539).

### LuaJIT

* LuaJIT now can handle exceptions on traces (gh-7745).

### Lua

* **[Breaking change]** Added the new `varbinary` type to Lua. An object of
  this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so
  it can be used for storing binary blobs in the database. This also works the
  other way round: data fields stored as `MP_BIN` are now decoded in Lua as
  varbinary objects, not as plain strings, as they used to be. Since the latter
  may cause compatibility issues, the new compat option `binary_data_decoding`
  was introduced to revert the built-in decoder to the old behavior (gh-1629).
* Introduced the new Lua module `trigger`. This module enables managing and
  calling triggers stored in trigger registry (gh-8656).
* Triggers from `space_object`, `box.session`, and `box.ctl` were moved to
  the trigger registry (gh-8657).
* Introduced a new event `'tarantool.trigger.on_change'` in the trigger registry.
  It is called when any event in the trigger registry is modified (gh-8664).
* Introduced helpers for encoding and decoding IPROTO packets in Lua (gh-8054).

#### Msgpack

* Improved error reporting for `msgpack.decode`. Now, an error raised by
  `mgpack.decode` has a detailed error message and the offset in the input
  data. If `msgpack.decode` failed to unpack a MsgPack extension, it also
  includes the error cause pointing to the error in the extension data
  (gh-7986).

#### Netbox

* Introduced the new `net.box` module function `from_fd` for creating a new
  connection from a socket file descriptor number (gh-8984).

#### Popen

* Introduced new option `inherit_fds` for `popen.new`. The option takes
  an array of file descriptor numbers that should be left open in the child
  process (gh-8926).

#### Socket

* Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and
  `socket:detach` (gh-8927).

#### Xlog

* Introduced the `xlog.meta()` method for reading a meta block from an xlog
  file.

### SQL

* Introduced the `SHOW CREATE TABLE` statement (gh-8098).
* **[Breaking change]** The `DROP CONSTRAINT` statement has been
  improved (gh-9112).
* **[Breaking change]** Names in SQL are now case sensitive. To support backward
  compatibility, a second lookup using a name normalized using the old rules is
  added (gh-4467).
* `SHOW CREATE TABLE` no longer supports the DEFAULT clause (gh-8793).
* `SQL_EXPR` functions can now be set as a default value (gh-8793).
* A literal set as the default value can no longer have a `+` or `-` sign unless
  the literal is numeric (gh-8793).
* **[Breaking change]** SQL now uses the `box` mechanism for default
  values (gh-8793).

### App

* Added the new `--force-recovery` CLI option (gh-8876).

### Box

* Non-system spaces in a snapshot file are now sorted by their IDs.
  As before, all non-system spaces are stored after system spaces (gh-7954).
* **[Breaking change]** Dropped support for `IPROTO_CALL16` from `net.box`
  (i.e., the `call_16` option and the `call_16` method for `net.box`
  connections) (gh-8636).
* Added support for square brackets in procedure resolution for Lua calls
  (gh-8604).
* Added support for accepting IPROTO requests with a space or index name instead
  of an identifier (gh-8146).
* Added a new `box.info` parameter `hostname` (gh-8605).
* Added the `box.cfg.memtx_sort_threads` parameter that specifies the number of
  threads used to sort indexes keys on loading a memtx database. OpenMP is
  not used to sort keys anymore (gh-3389).
* Changed the output of the `fselect` index method from a table of strings to
  a multiline string and dropped the `print` and `use_nbsp` options. With the
  `yaml_pretty_multiline` compat option enabled by default, multiline strings
  now look good in the console, so there's no need to return a table of strings
  to prettify the `fselect` output anymore.
* Allowed multiple updates to the same tuple field in a single call (gh-8226).
* Introduced `box.tuple.format` that enables format definition for tuples
  created via `box.tuple.new` (standalone tuples) (gh-4693).
* **[Breaking change]** Disabled argument list syntax of `box.tuple.new` (this
  was needed for gh-4693). It is possible to switch to the old behavior using
  the compatibility option `box_tuple_new_vararg`.
* Sped up `index.select` and `index.pairs` with the `after` option by up to 30%
  in a synthetic test by eliminating an extra buffer allocation.
* The function `box.session.push` is now deprecated. Consider using
  `box.broadcast` instead (gh-8802).
* Introduced the new function `box.session.new` for creating a new IPROTO
  session from a socket file descriptor number (gh-8801).
* Introduced the new `lua_eval` and `lua_call` object types for
  `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval`
  allows the user to execute an arbitrary Lua expression with the
  `IPROTO_EVAL` request. Granting the `'execute'` privilege on `lua_call`
  allows the user to execute any global user-defined Lua function with
  the `IPROTO_CALL` request (gh-8803).
* **[Breaking change]** Introduced the new `sql` object type for
  `box.schema.user.grant`. Now only users with the `'execute'` privilege
  granted on `sql` or `universe` can execute SQL expressions with the
  `IPROTO_EXECUTE` or `IPROTO_PREPARE` requests. To revert to the old behavior
  (no SQL access checks), use the `sql_priv` compat option (gh-8803).
* The maximum length of `box.cfg{}` string parameters is now 512 instead of 256.
* Added a new flag `tarantool.build.asan` that shows whether build
  flag `ENABLE_ASAN` is set.
* Added the `leader_name` field to `box.info.election` (gh-8931).
* Now it is possible to specify the sort order for each part of an index
  (gh-5529).
* Added support for sending tuple formats in IPROTO responses. Added a
  `box_tuple_extension` backward compatibility option to disable sending
  tuple formats in responses to IPROTO_CALL and IPROTO_EVAL requests (gh-8146).
* Added support for sending tuple formats in IPROTO_CALL and IPROTO_EVAL request
  arguments. `box_tuple_extension` backward compatibility option can be used to
  disable receiving tuple formats in IPROTO_CALL and IPROTO_EVAL request arguments
  (gh-8633).

### CLI

* **[Behavior change]** Disabled starting the Lua REPL by default when running
  Tarantool. Now, Tarantool yields the message that shows the command usage.
  To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via
  `stdin`, use dash (`-`) as the script name. For more information see a help
  message by running `tarantool -h` (gh-8613).
* Introduced new command-line options `--name` and `--config` (gh-8613).

### Compat

* The following compatibility options' defaults were switched to new behavior:
    * `yaml_pretty_multiline`
    * `sql_seq_scan_default`
    * `json_escape_forward_slash`
    * `fiber_channel_close_mode`
    * `fiber_slice_default`
    * `box_cfg_replication_sync_timeout`
    * `c_func_iproto_multireturn`

  More information on the new behavior can be found on the [Module compat](https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/) page.

### Config

* Initial version of the declarative server and cluster configuration
  (gh-8724).
* All vinyl options are now supported (gh-8861).
* All feedback options are now supported (gh-8861).
* The `memtx.sort_threads` option is now supported (gh-8861).
* The `bootstrap_leader` option is now supported (gh-8861).
* All flight recorder options are now supported (gh-8861).
* All security hardening `box.cfg{}` options are now supported by
  config (gh-8861).
* Added a low priority environment configuration source, which looks into
  `TT_*_DEFAULT` variables. It is useful to declare defaults (gh-8862).
* It is now possible to access configuration from the application script using
  the `config:get()` method (gh-8862).
* All metrics configuration options from `box.cfg{}` are now supported
  in the YAML config (gh-8861).
* Added `--help-env-list` CLI option (gh-8862).
* Introduced a non-public API for extending the declarative config
  functionality in Tarantool Community Edition (gh-8862).
* Improved the credentials applier: now it supports two-way synchronization
  of roles and privileges for both users and roles (gh-8861).
* Removed the sha1 and sha256 hash entries from the config credentials
  schema as not planned for development (gh-8967).
* Implemented a full password support in the `config.credentials` schema,
  including a password setting, updating and removal for the `chap-sha1`
  auth type (supported by both Tarantool Community Edition and Tarantool
  Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition
  where it is available) (gh-8967).
* A group name from a topology defined by the cluster configuration doesn't
  persisted anymore as `box.cfg.cluster_name` (gh-8862).
* The default directory and file paths are changed to `var/run/<...>`,
  `var/log/<...>`, `var/lib/<...>` and so on (gh-8862).
* The directory that contains the configuration file is now added into the Lua
  modules search paths (gh-8862).
* Most of vshard options are now added in the config (gh-9007).
* Added initial support of vshard (gh-9007).
* All audit options are now supported (gh-8861).
* Non-default privileges are now revoked from default users and roles
  when they are removed from the config (gh-8967).
* Introduced the initial support for roles - programs that run when
  a configuration is loaded or reloaded (gh-9078).
* On read-only instances, Tarantool now synchronizes credentials
  in the background when switching to read-write mode instead of
  skipping the synchronization (gh-8967).
* Now the permissions for space, function, and sequence can be granted
  via the config. If the object has not been created yet, the grant is
  postponed and executed after the creation of the object (gh-8967).
* Now the brand-new `lua_eval`, `lua_call`, and `sql` object types are
  supported by credentials (gh-8967).
* Now a password hash (and salt) will be regenerated for users managed
  in the configuration file if `security.auth_type` differs from a user's
  `auth_type` (gh-8967).
* Introduced validation for replicaset_name/uuid and instance_name/uuid
  mismatches before the recovery process when Tarantool is configured via
  a YAML file or etcd.
* If Tarantool is configured via a YAML file or etcd, then instance and replica
  set names are automatically set when possible (gh-8978).
* Added support for the `{{ replicaset_name }}` and `{{ group_name }}`
  templates in addition to the existing `{{ instance_name }}` (gh-8862).

### Console

* Now multiline commands can be used by setting continuation marker
  using the `\set continuation` command.

### Debugger

* Introduced readline support to tarantool debugger (gh-7738).

### Fiber

* Exported `fiber_set_name_n`, `fiber_name`, `fiber_id`, `fiber_csw` and
  `fiber_find` into the public C API and usable via FFI as well.
* Make `fiber_set_joinable`, `fiber_set_ctx` and `fiber_get_ctx`
  treat the NULL argument as the current fiber.

### Space

* Introduced space type: a new space definition field.
* Introduced the fully temporary space type. It is the same as data-temporary
  but also has temporary metadata. Temporary spaces can now be created in
  read-only mode, they disappear after server restart and don't exist on
  replicas (gh-8323).

### Test

#### Fuzz

* Added a grammar-based SQL fuzzer (gh-4826).

### Build

* Tarantool does not depend on libgomp anymore (gh-7689).
* Added building static `deb` and `rpm` packages.
* Dropped building distro-specific `deb` and `rpm` Tarantool packages.
* Dropped building Tarantool packages with GC32. Now only GC64 packages
  are shipped.
* Exported all LuaJIT public functions from Tarantool binary (gh-3680).
* Updated libcurl to version 8.3.0.

### Misc

* Updated luarocks version to 3.9.2 (gh-6597).

## Bugs fixed

### Core

* **[Breaking change]** The key `box.space._schema['cluster']` is renamed to
  `'replicaset_uuid'`. That is not expected to be breaking because `_schema` is
  an internal system space, but the key was visible in public and documented
  (gh-5029).
* **[Breaking change]** The table `box.info.cluster` is renamed to
  `box.info.replicaset`. The behaviour can be reverted using the `compat` option
  `box_info_cluster_meaning`
  (https://tarantool.io/compat/box_info_cluster_meaning) (gh-5029).
* Eliminated implicit conversion of unprintable utf-8 strings to binary blobs
  when encoded in YAML. Now unprintable characters are encoded as escaped utf-8
  code points, for example, `\x80` or `\u200B` (gh-8756).
* Fixed decoding of escape sequences for single-byte character codes from YAML.
  Before the fix, single-byte character codes between `0x80` and `0xff` would
  be erroneously converted to two-byte UTF-8 code points, for example, `\x80`
  would be decoded as `\uC280` (gh-8782).
* Now `index_object.parts.exclude_null` always contains `false` rather than
  `null` when it is actually `false` (gh-8649).
* Fixed a crash that could happen when inserting an index definition with an
  empty parts list directly into `box.space._index` (gh-8688).
* Fixed the memory leak on unpacking an invalid MsgPack error extension
  (gh-8921).
* Fixed a use-after-free bug in fiber recycling code (gh-9020).
* Fixed a use-after-free bug in iproto server code (gh-9037).
* Fixed a heap-buffer-overflow bug in fiber creation code (gh-9026).
* An error is no longer raised on an attempt to print a dead fiber (gh-4265).
* Fixed a possible out-of-bound write in debug build on unpacking a MsgPack
  error extension (gh-9098).
- Fixed a bug that caused a replication error after calling
  `box.schema.downgrade` (gh-9049).
* Fixed a possible crash when unpacking an invalid MsgPack error extension
  (gh-9136).
* Fixed a wrong assertion in index comparators when comparing decimals with
  floats greater than `1e38`. The error was present only in the debug build.
  Despite the failing assertion, the behavior after the assertion was correct
  (gh-8472).
* Now the "Type Ctrl-C to exit Tarantool" message is displayed when a local
  console is exited, but background fibers are running (gh-7017).
* Fixed a bug that could result in the incorrect `space:bsize()` when altering
  a primary index concurrently with DML operations (gh-9247).
* Fixed a crash that happened while printing the stack trace on a stack
  overflow bug (gh-9222).
* Fixed the behavior of `log.cfg{modules = ...}`. Now, instead of merging the
  new log modules configuration with the old one, it completely overwrites the
  current configuration, which is consistent with `box.cfg{log_modules = ...}`
  (gh-7962).
* Fixed a bug in the network buffer that could result in a crash when there are
  a lot of pending requests (gh-9218)
* Fixed a potential data corruption when the number of tuples in a hash index
  is more than 2147483648 (gh-3594).
* Decremented the max space id (`box.schema.SPACE_MAX`). Now, the max space id
  equals 2147483646. The limit was decremented because the old value is used as
  an error indicator in the box C API. It's still possible to revert to the old
  behavior with the compatibility module option `box_space_max` (gh-9118).
* The `tarantool -v` output and `tarantool.build.flags` string now include
  build type-specific and LTO flags if any (gh-8022).
* Fixed a possible inconsistent state entering if fibers are joined incorrectly.
  Now the `fiber_set_joinable` function panics if the fiber is dead or joined
  already. The `fiber_join` and `fiber_join_timeout` functions now panic on a
  double join if it is possible to detect it (gh-7562).

### Replication

* Fixed a bug that allowed to change replicaset UUID via `_schema` space.
* Fixed a bug when new instances could try to register via an anon instance
  which previously failed to apply `box.cfg{replication_anon = false}`.
* Fixed a bug when `box.ctl.demote()` with `box.cfg{election_mode = 'off'}`
  and an owned synchro queue could simply not do anything (gh-6860).
* Fixed a false-positive split-brain in a replica set on the first
  promotion after an upgrade from versions before 2.10.1 (gh-8996).
* Fixed replicas writing corrupted xlogs when appending data to a local space
  from an `on_replace` or `before_replace` trigger on a global replicated space.
  Such xlogs were unrecoverable and caused other nodes to break replication with
  the replica (gh-8746, gh-8958).

### LuaJIT

Backported patches from the vanilla LuaJIT trunk (gh-8825). The following issues
were fixed as part of this activity:

* Fixed `BC_UCLO` insertion for returns.
* Fixed recording of `BC_VARG` with unused vararg values.
* Initialization instructions on trace are now emitted only for the first
  member of a union.
* Prevent integer overflow while parsing long strings.
* Fixed various `^` operator and `math.pow()` function inconsistencies.
* Fixed parsing with predicting `next()` and `pairs()`.
* Fixed binary number literal parsing. Parsing of binary number with a zero
  fractional part raises error too now.
* Fixed load forwarding optimization applied after table rehashing.
* Fixed recording of the `BC_TSETM`.
* Fixed the panic routine when `mprotect` fails to change flags for mcode area.
* Fixed frame for on-trace OOM handling.
* Fixed handling of instable types in TNEW/TDUP load forwarding.
* Handled table unsinking in the presence of `IRFL_TAB_NOMM`.
* Fixed a bug when an error could be raised on the non-currently executed
  coroutine (gh-6323).
Backported patches from the vanilla LuaJIT trunk (gh-9145). The following issues
were fixed as part of this activity:

* Fixed error handling after return from a child coroutine.

### Lua

#### Fio

* The default permission mode for `fio.open()` was changed for newly
  created files to 0666 (before umask) (gh-7981).

### SQL

* Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause
  (ghs-119).
* Fixed a memory leak in a parser for a column's default rule (gh-9159).
* Fixed assertion in a debug build when a collation was added after an index
  with more than one field (gh-9229).
* Fixed a bug that in some cases would not assign collations to an index created
  during `CREATE TABLE` (gh-9229).
* A space no longer loses a default value after
  `ALTER TABLE ADD COLUMN` (gh-8793).

### Box

- Fixed the ability to drop `_vinyl_deferred_delete` system space (gh-5279).
* Fixed a bug when it was not possible to access spaces named `_stream` and
  `_stream_space_cache` and indexes named `_space` using the `net.box` stream
  objects (gh-8598).
* **[Breaking change]** The `cord_slab_cache` symbol was removed from the public
  API export (gh-7124).
* All DDL functions from the `box.schema` module are now wrapped into a
  transaction to avoid database inconsistency on failed operations (gh-4348).
* Fixed crashes if `box.info.memory()`, `box.info.gc()`, `box.info.vinyl()`,
  and `box.info.sql()` are called before `box.cfg{}` (gh-9173).
* **[Breaking change]** Added a `c_func_iproto_multireturn` option to the
  `compat` module. The new behavior drops an additional array that wraps
  multiple results returned via iproto (gh-4799).
* Fixed a bug that allows downgrading from a schema version more recent than
  a Tarantool version (gh-9182).
* Fixed a bug when `on_shutdown` triggers weren't run if `os.exit()` was
  called from `-e` command-line option (gh-9266).
* **[Breaking change]** `box.schema.user.grant()` now raises an error on
  an attempt to grant the `execute` privilege on a space.  Historically,
  this action was allowed although it had no effect. It's still possible
  to revert to the old behavior with the new compatibility option
  `box_space_execute_priv` (gh-9277).
* Added a check that disables setting `box.cfg` and `log.cfg` options to
  infinite numbers (NaN, Inf). Setting a `box.cfg` or `log.cfg` option to
  an infinite number could result in a crash or invalid behavior (gh-4962).

### Config

* Fixed an error message if the cluster configuration was not provided or the
  instance was not found in the cluster configuration during reload (gh-8862).
* Support parent directories creation for options that accept a directory or a
  file (gh-8862).
* Create parent directories for `console.socket` and `log.file` (gh-8862).
* Create the `process.work_dir` directory (gh-8862).
* Consider all the paths as relative to `process.work_dir` when creating
  necessary directories (gh-8862).
* Added the validation of instance, replica set, and group names in the
  configuration, `--name` CLI option, and `TT_INSTANCE_NAME` environment
  variable (gh-8862).

### Http

* Fixed a streaming connection stuck if etcd is stopped
  unexpectedly (gh-9086).
