07 Feb 14:22

systay

1bdb462

Vitess v19.0.0-rc1 Pre-release

Pre-release

Release of Vitess v19.0.0

Summary

Major Changes
Minor Changes
- Apply VSchema
  - --strict sub-flag and strict gRPC field

Major Changes

Dropping Support for MySQL 5.7

Oracle has marked MySQL 5.7 end of life as of October 2023. Vitess is also dropping support for MySQL 5.7 from v19 onwards. Users are advised to upgrade to MySQL 8.0 while on v18 version of Vitess before
upgrading to v19.

Vitess will however, continue to support importing from MySQL 5.7 into Vitess even in v19.

Deprecations and Deletions

The MYSQL_FLAVOR environment variable is now removed from all Docker Images.

VTTablet Flags

The following flags — which were deprecated in Vitess 7.0 — have been removed:
--vreplication_healthcheck_topology_refresh, --vreplication_healthcheck_retry_delay, and --vreplication_healthcheck_timeout.
The --vreplication_tablet_type flag is now deprecated and ignored.

Docker Image vitess/lite

The mysqld binary is now deprecated in the vitess/lite Docker image and will be removed in a future release.
This means that the MySQL/Percona version specific image tags for the vitess/lite image are deprecated.

Below is a full list of available tags for v19.0.0 and their deprecation status:

Image	Deprecated
`vitess/lite:v19.0.0`	NO
`vitess/lite:v19.0.0-mysql57`	YES
`vitess/lite:v19.0.0-mysql80`	YES
`vitess/lite:v19.0.0-percona57`	YES
`vitess/lite:v19.0.0-percona80`	YES

If you are currently using vitess/lite as your mysqld image in your vitess-operator deployment we invite you to use an official MySQL image, such as mysql:8.0.30.

Below is an example of a kubernetes yaml file before and after upgrading to an official MySQL image:

# before:

# the image used here includes MySQL 8.0.30 and its binaries

    mysqld:
      mysql80Compatible: vitess/lite:v19.0.0-mysql80

# after:

# if we still want to use MySQL 8.0.30, we now have to use the
# official MySQL image with the 8.0.30 tag as shown below 

    mysqld:
      mysql80Compatible: mysql:8.0.30 # or even mysql:8.0.34 for instance

Explain Statement Format

Explain statement format vitess and vexplain were deprecated in v16 and removed in v19 version.
Use VExplain Statement for understanding Vitess plans.

Breaking Changes

ExecuteFetchAsDBA rejects multi-statement SQL

vtctldclient ExecuteFetchAsDBA (and similarly the vtctl and vtctlclient commands) now reject multi-statement SQL with error.

For example, vtctldclient ExecuteFetchAsDBA my-tablet "stop replica; change replication source to auto_position=1; start replica will return an error, without attempting to execute any of these queries.

Previously, ExecuteFetchAsDBA silently accepted multi statement SQL. It would (attempt to) execute all of them, but:

It would only indicate error for the first statement. Errors on 2nd, 3rd, ... statements were silently ignored.
It would not consume the result sets of the 2nd, 3rd, ... statements. It would then return the used connection to the pool in a dirty state. Any further query that happens to take that connection out of the pool could get unexpected results.
As another side effect, multi-statement schema changes would cause schema to be reloaded with only the first change, leaving the cached schema inconsistent with the underlying database.

ExecuteFetchAsDBA does allow a specific use case of multi-statement SQL, which is where all statements are in the form of CREATE TABLE or CREATE VIEW. This is to support a common pattern of schema initialization, formalized in ApplySchema --batch-size which uses ExecuteFetchAsDBA under the hood.

New Stats

Stream Consolidations

Prior to 19.0 VTTablet reported how much time non-streaming executions spend waiting for consolidations to occur. In 19.0, VTTablet reports a similar stat for streaming executions in /debug/vars stat Waits.Histograms.StreamConsolidations.

Build Version in `/debug/vars`

The build version (e.g., 19.0.0-SNAPSHOT) has been added to /debug/vars, allowing users to programmatically inspect Vitess components' build version at runtime.

Planned Reparent Shard

`--tolerable-replication-lag` Sub-flag

A new sub-flag --tolerable-replication-lag has been added to the command PlannedReparentShard that allows users to specify the amount of replication lag that is considered acceptable for a tablet to be eligible for promotion when Vitess makes the choice of a new primary.
This feature is opt-in and not specifying this sub-flag makes Vitess ignore the replication lag entirely.

A new flag in VTOrc with the same name has been added to control the behaviour of the PlannedReparentShard calls that VTOrc issues.

Query Compatibility

Multi Table Delete Support

Support is added for sharded multi-table delete with target on single table using multiple table join.

Example: Delete t1 from t1 join t2 on t1.id = t2.id join t3 on t1.col = t3.col where t3.foo = 5 and t2.bar = 7

More details about how it works is available in MySQL Docs

`SHOW VSCHEMA KEYSPACES` Query

A SQL query, SHOW VSCHEMA KEYSPACES is now supported in Vitess. This query prints the vschema information
for all the keyspaces. It is useful for seeing the foreign key mode, whether the keyspace is sharded, and if there is an
error in the VSchema for the keyspace.

An example output of the query looks like -

mysql> show vschema keyspaces;
+----------+---------+-------------+---------+
| Keyspace | Sharded | Foreign Key | Comment |
+----------+---------+-------------+---------+
| ks       | true    | managed     |         |
| uks      | false   | managed     |         |
+----------+---------+-------------+---------+
2 rows in set (0.01 sec)

`FOREIGN_KEY_CHECKS` is now a Vitess Aware Variable

When VTGate receives a query to change the FOREIGN_KEY_CHECKS value for a session, instead of sending the value down to MySQL, VTGate now keeps track of the value and changes the queries by adding SET_VAR(FOREIGN_KEY_CHECKS=On/Off) style query optimizer hints wherever required.

Explain Statement

Explain statement can handle routed table queries now. Explain is unsupported when the tables involved in the query refers more than one keyspace. Users should use VExplain Statement in those cases.

Partial Multi-shard Commit Warnings

When using multi transaction mode (the default), it is possible for Vitess to successfully commit to one shard, but fail to commit to a subsequent shard, thus breaking the atomicity of a multi-shard transaction.

In v19.0, VTGate reports partial-success commits in warnings, e.g.:

mysql> commit;
ERROR 1317 (70100): target: customer.-80.primary: vttablet: rpc error: code = Aborted desc = transaction 1703182545849001001: ended at 2023-12-21 14:07:41.515 EST (exceeded timeout: 30s) (CallerID: userData1)
mysql> show warnings;
+---------+------+----------------------------------------------------------+
| Level   | Code | Message                                                  |
+---------+------+----------------------------------------------------------+
| Warning |  301 | multi-db commit failed after committing to 1 shards: 80- |
+---------+------+----------------------------------------------------------+
1 row in set, 1 warning (0.00 sec)

Vttestserver

`--vtcombo-bind-host` flag

A new flag --vtcombo-bind-host has been added to vttestserver that allows the us...

Contributors

arthurschreiber, dbussink, and 48 other contributors

shlomi-noach and harshit-gangal

Assets 5

06 Nov 12:13

harshit-gangal

v18.0.0

9a6f526

Vitess v18.0.0

Release of Vitess v18.0.0

Summary

Major Changes

Major Changes

Breaking Changes

Local examples now use etcd v3 storage and API

In previous releases the local examples were
explicitly using etcd v2 storage (etcd --enable-v2=true) and API (ETCDCTL_API=2) mode. We have now
removed this legacy etcd usage and instead use the new (default) etcd v3 storage and API. Please see
PR #13791 for details. If you are using the local
examples in any sort of long-term non-testing capacity, then you will need to explicitly use the v2 storage
and API mode or migrate your existing data from v2 to v3.

New command line flags and behavior

VTOrc flag `--allow-emergency-reparent`

VTOrc has a new flag --allow-emergency-reparent that specifies whether VTOrc is allowed to run emergency
failover operations. Users that want VTOrc to fix replication issues, but don't want it to run any failovers
should use this flag. This flag defaults to true which corresponds to the default behavior from prior releases.

VTOrc flag `--change-tablets-with-errant-gtid-to-drained`

VTOrc has a new flag --change-tablets-with-errant-gtid-to-drained that allows users to choose whether VTOrc should change the
tablet type of tablets with errant GTIDs to DRAINED. By default, this flag is disabled.

This feature allows users to configure VTOrc such that any tablet that encounters errant GTIDs is automatically taken out of the
serving graph. These tablets can then be inspected for what the errant GTIDs are, and once fixed, they can rejoin the cluster.

ERS sub flag `--wait-for-all-tablets`

vtctldclient command EmergencyReparentShard has a new sub-flag --wait-for-all-tablets that makes EmergencyReparentShard wait
for a response from all the tablets. Originally EmergencyReparentShard was meant only to be run when a primary tablet is unreachable.
We have realized now that there are cases when replication is broken but all tablets are reachable. In these cases, it is advisable to
call EmergencyReparentShard with --wait-for-all-tablets so that it does not ignore any of the tablets.

VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming`

This flag enables transaction support on VTGate's StreamExecute gRPC API.
When this is enabled, StreamExecute will return the session in the last packet of the response.
Users should enable this flag only after client code has been changed to expect such a packet.

It is disabled by default.

Experimental Foreign Key Support

A new optional field foreignKeyMode has been added to the VSchema. This field can be provided for each keyspace. The VTGate flag --foreign_key_mode has been deprecated in favor of this field.

There are 3 foreign key modes now supported in Vitess -

unmanaged -
This mode represents the default behavior in Vitess, where it does not manage foreign key column references. Users are responsible for configuring foreign keys in MySQL in such a way that related rows, as determined by foreign keys, reside within the same shard.
managed [EXPERIMENTAL] -
In this experimental mode, Vitess is fully aware of foreign key relationships and actively tracks foreign key constraints using the schema tracker. VTGate will handle DML operations with foreign keys and correctly cascade updates and deletes.
It will also verify restrict constraints and validate the existence of parent rows before inserting child rows.
This ensures that all child operations are logged in binary logs, unlike the InnoDB implementation of foreign keys.
This allows the usage of VReplication workflows with foreign keys.
Implementation details are documented in the RFC for foreign keys.
disallow -
In this mode Vitess explicitly disallows any DDL statements that try to create a foreign key constraint. This mode is equivalent to running VTGate with the flag --foreign_key_mode=disallow.

In addition to query support, there is a new flag to MoveTables called --atomic-copy which should be used to import data into Vitess from databases which have foreign keys defined in the schema.

Upgrade process

After upgrading from v17 to v18, users should specify the correct foreign key mode for all their keyspaces in the VSchema using the new property.
Once this change has taken effect, the deprecated flag --foreign_key_mode can be dropped from all VTGates. Note that this is only required if running in disallow mode.
No action is needed to use unmanaged mode.

VTAdmin

vtadmin-web updated to node v18.16.0 (LTS)

Building vtadmin-web now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed
in https://nodejs.org/en/blog/release/v18, but none apply to VTAdmin. Full details on node v18.16.0 are listed
on https://nodejs.org/en/blog/release/v18.16.0.

Deprecations and Deletions

Legacy Client Binaries

vtctldclient is our new modern Vitess controller daemon (vtctld) client – which you will use to perform commands
and take actions in your Vitess clusters. It is replacing the legacy vtctl/vtctlclient binaries.
Some of the benefits are:

Dedicated RPCs for each command
that are used between vtctldclient and vtctld – this offers clean separation of commands and makes it easier to
develop new features without impacting other commands. This also presents an API that other clients (both Vitess and
3rd party) can use to interface with Vitess.
Use of modern frameworks: pFlag, Cobra, and Viper.
This makes development easier while also offering a better UX. For example, this offers a way to use
configuration files with support for
dynamic configuration (see also).
The reference documentation is now built through code. This
removes a burden from developers while helping users by ensuring the docs are always correct and up-to-date.

In Vitess 18 we have completed migrating all client commands to vtctldclient – the last ones being the OnlineDDL
and VReplication commands. With this work now completed, the
legacy vtctl/vtctlclient binaries are now fully deprecated and we plan to remove them in Vitess 19. You should
begin your transition before upgrading to Vitess 18.

Deprecated Command Line Flags

Throttler related vttablet flags:

--throttle_threshold is deprecated and will be removed in v19
--throttle_metrics_query is deprecated and will be removed in v19
--throttle_metrics_threshold is deprecated and will be removed in v19
--throttle_check_as_check_self is deprecated and will be removed in v19
--throttler-config-via-topo is deprecated after assu...

Contributors

dbussink, vmg, and 30 other contributors

Assets 5

03 Oct 14:24

harshit-gangal

v18.0.0-rc1

6ab165a

Vitess v18.0.0-rc1 Pre-release

Pre-release

Release of Vitess v18.0.0-rc1

Summary

Major Changes

Major Changes

Breaking Changes

Local examples now use etcd v3 storage and API

In previous releases the local examples were
explicitly using etcd v2 storage (etcd --enable-v2=true) and API (ETCDCTL_API=2) mode. We have now
removed this legacy etcd usage and instead use the new (default) etcd v3 storage and API. Please see
PR #13791 for additional info. If you are using the local
examples in any sort of long-term non-testing capacity, then you will need to explicitly use the v2 storage
and API mode or migrate your existing data from v2 to v3.

New command line flags and behavior

VTOrc flag `--allow-emergency-reparent`

VTOrc has a new flag --allow-emergency-reparent that allows the users to toggle the ability of VTOrc to run emergency
reparent operations. Users that want VTOrc to fix the replication issues, but don't want it to run any reparents
should start using this flag. By default, VTOrc will be able to run EmergencyReparentShard. Users must specify the
flag to false to change the behaviour.

VTOrc flag `--change-tablets-with-errant-gtid-to-drained`

ERS sub flag `--wait-for-all-tablets`

Running EmergencyReparentShard from the vtctldclient has a new sub-flag --wait-for-all-tablets that makes EmergencyReparentShard wait
for a response from all the tablets. Originally EmergencyReparentShard was meant only to be run when a primary tablet is unreachable.
We have realized now that there are cases when the replication is broken but all the tablets are reachable. In these cases, it is advisable to
call EmergencyReparentShard with --wait-for-all-tablets so that it does not ignore one of the tablets.

VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming`

This flag enables transaction support on StreamExecute api.
Once enabled, VTGate StreamExecute gRPC api will send session as the last packet in the response.
The client should enable it only when they have made the required changes to expect such a packet.

It is disabled by default.

Experimental Foreign Key Support

A new field foreignKeyMode has been added to the VSchema. This field can be provided for each keyspace. The VTGate flag --foreign_key_mode has been deprecated in favour of this field.

There are 3 foreign key modes now supported in Vitess -

unmanaged -
This mode represents the default behaviour in Vitess, where it does not manage foreign keys column references. Users are responsible for configuring foreign keys in MySQL in such a way that related rows, as determined by foreign keys, reside within the same shard.
managed [EXPERIMENTAL] -
In this experimental mode, Vitess is fully aware of foreign key relationships and actively tracks foreign key constraints using the schema tracker. Vitess takes charge of handling DML operations with foreign keys cascading updates, deletes and verifying restrict. It will also validate parent row existence.
This ensures that all the operations are logged in binary logs, unlike MySQL implementation of foreign keys.
This enables seamless integration of VReplication with foreign keys.
For more details on what operations Vitess takes please refer to the design document for foreign keys.
disallow -
In this mode Vitess explicitly disallows any DDL statements that try to create a foreign key constraint. This mode is equivalent to running VTGate with the flag --foreign_key_mode=disallow.

Upgrade process

After upgrading from v17 to v18, the users should specify the correct foreign key mode for all their keyspaces in the VSchema using the new property.
Once this change has taken effect, the deprecated flag --foreign_key_mode can be dropped from all the VTGates.

VTAdmin

vtadmin-web updated to node v18.16.0 (LTS)

Building vtadmin-web now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed
in https://nodejs.org/en/blog/release/v18.0.0, but none apply to VTAdmin. Full details on v18.16.0 are listed
on https://nodejs.org/en/blog/release/v18.16.0.

Deprecations and Deletions

Deprecated Command Line Flags

Throttler related vttablet flags:

--throttle_threshold is deprecated and will be removed in v19.0
--throttle_metrics_query is deprecated and will be removed in v19.0
--throttle_metrics_threshold is deprecated and will be removed in v19.0
--throttle_check_as_check_self is deprecated and will be removed in v19.0
--throttler-config-via-topo is deprecated after assumed true in v17.0. It will be removed in a future version.

Cache related vttablet flags:

--queryserver-config-query-cache-lfu is deprecated and will be removed in v19.0. The query cache always uses a LFU implementation now.
--queryserver-config-query-cache-size is deprecated and will be removed in v19.0. This option only applied to LRU caches, which are now unsupported.

Buffering related vtgate flags:

--buffer_implementation is deprecated and will be removed in v19.0

Cache related vtgate flags:

--gate_query_cache_lfu is deprecated and will be removed in v19.0. The query cache always uses a LFU implementation now.
--gate_query_cache_size is deprecated and will be removed in v19.0. This option only applied to LRU caches, which are now unsupported.

VTGate flags:

--schema_change_signal_user is deprecated and will be removed in v19.0
--foreign_key_mode is deprecated and will be removed in v19.0. For more detail read the foreign keys section.

VDiff v1:

VDiff v2 was added in Vitess 15.0 and marked as GA in 16.0.
The legacy v1 client command is now deprecated in Vitess 18.0 and will be removed in 19.0.
Please switch all of your usage to the new VDiff client command ASAP.

Deprecated Stats

The following EmergencyReparentShard stats are deprecated in v18.0 and will be removed in v19.0:

ers_counter
ers_success_counter
ers_failure_counter

These metrics are replaced by new reparenting stats introduced in v18.0.

VTBackup stat DurationByPhase is deprecated. Use the binary-valued Phase stat instead.

Deleted Command Line Flags

Flags in vtcombo:

--vtctld_addr

Flags in vtctldclient ApplySchema:

--skip-preflight

Flags in vtctl ApplySchema:

--skip_preflight

Flags in vtgate:

--vtctld_addr

Flags in vttablet:

--vtctld_addr
--use_super_read_only
--disable-replication-manager
--init_populate_metadata
--queryserver-config-pool-prefill-parallelism
`-...

Contributors

dbussink, vmg, and 29 other contributors

Assets 5

03 Oct 14:42

mattlord

v17.0.3

dced85d

Vitess v17.0.3

What's Changed

Back to dev mode after v17.0.2 by @mattlord in #13815
[release-17.0] Fix BackupShard to get its options from its own flags (#13813) by @vitess-bot in #13820
[release-17.0] Flakes: empty vtdataroot before starting a new vreplication e2e test (#13803) by @vitess-bot in #13822
[release-17.0] Bump upgrade tests to go1.21.0 by @frouioui in #13855
[release-17.0] Use Debian Bullseye in Bootstrap by @frouioui in #13757
[release-17.0] OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations (#13928) by @vitess-bot in #13937
[release-17.0] Upgrade the Golang version to go1.20.8 by @github-actions in #13934
[release-17.0] VReplication: Handle SQL NULL and JSON 'null' correctly for JSON columns (#13944) by @vitess-bot in #13947
[release-17.0] Tablet throttler: empty list of probes on non-leader (#13926) by @vitess-bot in #13952
[release-17.0] VDiff: correct handling of default source and target cells (#13969) by @vitess-bot in #13984
[release-17.0] copy over existing vreplication rows copied to local counter if resuming from another tablet (#13949) by @vitess-bot in #13963
[release-17.0] Make Static Code Checks Etc fail if the ./changelog folder is out-of-date (#14003) by @vitess-bot in #14006
[release-17.0] fix data race in join engine primitive olap streaming mode execution (#14012) by @vitess-bot in #14016
[release-17.0] Enable failures in tools/e2e_test_race.sh and fix races (#13654) by @vitess-bot in #14011
[release-17.0] Flakes: skip flaky check that ETA for a VReplication VDiff2 Progress command is in the future. (#13804) by @vitess-bot in #13817
[release-17.0] Flakes: Synchronize access to logErrStacks in vterrors (#13827) by @vitess-bot in #13834
[release-17.0] fix: cost to include subshard opcode (#14023) by @vitess-bot in #14027
[release-17.0] moved timeout test to different package (#14028) by @vitess-bot in #14032
Backport [release-17.0] Flakes: Add recently added 'select rows_copied' query to ignore list #13993 by @rohit-nayak-ps in #14039
[release-17.0] TableGC: support DROP VIEW (#14020) by @vitess-bot in #14045
[release-17.0] OnlineDDL: cleanup cancelled migration artifacts; support --retain-artifacts=<duration> DDL strategy flag (#14029) by @vitess-bot in #14037
[release-17.0] handle large number of predicates without timing out (#13979) by @vitess-bot in #13982
[release-17.0] Add session flag for stream execute grpc api (#14046) by @vitess-bot in #14053
[release-17.0] backport mysqlctl CLI compatibility fix to 17.0 by @ajm188 in #14082
[release-17.0] evalengine: Mark UUID() function as non-constant (#14051) by @vitess-bot in #14057
[release-17.0] Rewrite USING to ON condition for joins (#13931) by @vitess-bot in #13941
[release-17.0] json: Fix quoting JSON keys (#14066) by @vitess-bot in #14068
[release-17.0] update docgen to embed commit ID in autogenerated doc frontmatter (#14056) by @vitess-bot in #14074
[release-17.0] anonymize homedirs in generated docs (#14101) by @vitess-bot in #14106
[release-17.0] VDiff: properly split cell values in record when using TabletPicker (#14099) by @vitess-bot in #14103
[release-17.0] bugfix: change column name and type to json (#14093) by @vitess-bot in #14117
[release-17.0] Remove FOSSA Test from CI until we can do it in a secure way (#14119) by @vitess-bot in #14122
[release-17.0] VDiff: Cleanup the controller for a VDiff before deleting it (#14107) by @vitess-bot in #14125
Code freeze of release-17.0 by @mattlord in #14138
[release-17.0] docker: add dedicated vtorc container (#14126) by @vitess-bot in #14147
[release-17.0] Fix upgrade tests by @frouioui in #14143
Release of v17.0.3 by @mattlord in #14140

Full Changelog: v0.17.2...v17.0.3

Contributors

mattlord, ajm188, and 3 other contributors

Assets 5

Releases: vitessio/vitess

Vitess v19.0.0-rc1

Release of Vitess v19.0.0

Summary

Table of Contents

Contributors

Vitess v18.0.2

Release of Vitess v18.0.2

Contributors

Vitess v17.0.5

Release of Vitess v17.0.5

Contributors

Vitess v16.0.7

Release of Vitess v16.0.7

Contributors

Vitess v18.0.1

Release of Vitess v18.0.1

Contributors

Vitess v17.0.4

Release of Vitess v17.0.4

Contributors

Vitess v16.0.6

Release of Vitess v16.0.6

Contributors

Vitess v18.0.0

Release of Vitess v18.0.0

Summary

Table of Contents

Upgrade process

Contributors

Vitess v18.0.0-rc1

Release of Vitess v18.0.0-rc1

Summary

Table of Contents

Upgrade process

Contributors

Vitess v17.0.3

What's Changed

Contributors